SlowGuess/NAACL_2021_OCR huggingface dataset:
lect questions for each of the  $N_0$  candidates. Let gen_interview計劃 denote the above optimization program for selecting questions for a given candidate. Output of this program is a Boolean vector sol, where  $\text{sol}(i) = 1$  if question  $q_i \in Q$  is to be included in the interview plan for the given candidate; 0 otherwise. We extend our earlier definition of a question bank by adding another property for each question viz., novelty count  $\beta(q_i)$ , which is the  $N_0 - \text{count}(q_i)$ , with count  $(q_i)$  being number of times the question  $q_i \in Q$  has been used so far. Initially  $\text{count}(q_i) = 0$  and  $\beta(q_i) = N_0$  for each question in  $Q$ . We sequentially call gen_interview計劃 for each of the  $N_0$  candidates and use the output Boolean vector sol to increment  $\beta(q_i)$  for each question in  $Q$ ; see algorithm gen_interview計劃_set.
The optimization program gen interviewing plan is same as earlier, except that we have added a new term  $f_{6}$  to the objective function that maximizes the sum of novelty counts of the questions.
$$
f _ {6}: \sum_ {i = 1} ^ {| Q |} x _ {i} \cdot \beta \left(q _ {i}\right) \tag {1}
$$
Again, we normalize this to ensure that the value is between 0 and 1. Note that the novelty counts are updated for all questions after each call to the optimization program.
Integer programming being an NP-Hard problem, exact solvers often take a lot of time. Instead, we resorted to the LP rounding approximation, where we relax the integer constraints  $x_{i} \in \{0,1\}$  to  $x_{i} \in [0,1]$  and used available LP solvers (CBC solver in python Pulp) to solve the resultant linear programming problem. Then we rounded the  $x_{i}$  values to  $\{0,1\}$  using a threshold.
4 Intrinsic Evaluation
While we do use human experts to compare two sets of questions, here we propose an automated intrinsic evaluation method to judge the quality of the set of questions (e.g., selected by our optimization model, or by any baseline method discussed later) for a particular candidate, by comparing this set with the set of questions actually asked to this candidate in a real interview. Let  $A_{i}$  denote the set of actual questions asked to  $i$ -th candidate, ignoring the order of asking; we can get  $A_{i}$  from interview transcripts. Let  $Q_{i}$  be the set of questions recommended by some algorithm for the  $i$ -th candidate. In general,  $|A_{i}| \neq |Q_{i}|$ . We assume that both  $Q_{i}$  and  $A_{i}$  are about the given skill  $s$ .
Suppose we have a Boolean function is_qsim (discussed shortly) that returns TRUE if two given questions are "highly similar" and FALSE otherwise. In forward evaluation, we compare each question  $q_{j} \in Q_{i}$  with questions in  $A_{i}$  and define a forward Boolean score such that  $b_{f}(q_{j}) = 1$  is there is at least one question  $a_{k} \in A_{i}$  such that is_qsim  $(q_{j},a_{k}) = TRUE$  and 0 otherwise. The quality of  $Q_{i}$  is evaluated based on the number of questions in  $Q_{i}$  having score 1.
$$
\operatorname {q u a l} _ {f} \left(Q _ {i}\right) = \frac {1}{\left| A _ {i} \right|} \cdot \sum_ {j = 1} ^ {\left| Q _ {i} \right|} b _ {f} \left(q _ {j}\right) \tag {2}
$$
Enforcing an additional constraint that a question in  $A_{i}$  is "matched" to at most one question in  $Q_{i}$ , ensures that score  $qual_{f}(Q_{i})$  is between 0 and 1.
In backward evaluation, we compare each question  $a_{j} \in A_{i}$  with questions in  $Q_{i}$  and define a backward Boolean score such that  $b_{b}(a_{j}) = 1$  if there is at least one question  $q_{k} \in Q_{i}$  such that  $is_q sim(q_k, a_j) = TRUE$  and 0 otherwise. The quality measure  $qual_{b}(A_{i})$  is defined analogously.
It remains now to define is_qsim. Several similarity measures have been designed in the literature for short text similarity. For efficiency, we consider a simple measure: two questions are sim
ilar if their coverages share at least a required minimum number of concepts  $k_{0}$  (which is a user-specified constant):
$$
i s _ {-} q s i m \left(q _ {i}, q _ {j}\right) = \left\{ \begin{array}{l l} T & \text {i f} | \psi \left(q _ {i}\right) \cap \psi \left(q _ {j}\right) | \geq k _ {0} \\ F & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
For example, questions 12 and 13 in Fig. 2 are similar (i.e., is_qsim(12, 13) = TRUE), assuming  $k_{0} = 3$ , since concepts 8, 10, 14 are common in  $\psi(12)$  and  $\psi(13)$ .
5 NLP for Interview Resource Creation
While natural language processing does not play a direct role in the optimization formulation for the selection of interview questions, it is crucial for the creation of the prerequisite resources.
The first task is the annotation of questions to identify concepts. These concepts are used to determine question coverage and to construct the question graph based on their coverage. We also use these annotations to construct the knowledge graph for skills, as we explain below. There is a rich body of literature in both mention detection and named entity disambiguation (Ferragina and Scaiella, 2012; Ganea and Hofmann, 2017; Sakor et al., 2020). Since we focus on skill-graphs which are largely sub-graphs of DBPedia, we use the publicly available APIs from TAGME (Ferragina and Scaiella, 2012). However, this resulted in two types of errors. Many mentions were annotated with concepts irrelevant for our skills. Secondly, many mentions relevant for our skills were left unannotated. We performed manual curation on the TAGME annotations to correct these two types of errors.
The second task is extraction of skills from resumes. We use an information extraction system called RINX (Pawar et al., 2017), which uses gazetteer-based, linguistic patterns based machine learning methods (e.g., CRF, BiLSTM) to extract mentions of various entity types and relations from resumes. For example, RINX extracts mentions of SKILL (e.g., Machine_learning, Python, CONCEPT (e.g., Activation function, Maximum margin), ROLE (e.g., Developer, DBA, Test_Engineer), TASK (e.g., Developed vendor master, Performance Tuning), among other entity types. The extracted skills are again annotated according to concepts by using TAGME, and manually curated to correct errors.
The next task is construction of the knowledge graph for different skills, based on the concept annotation of the questions and the extracted resume skills. This problem has not received enough attention in the computational linguistics community. We first identified a subgraph of DBPedia (Faralli et al., 2018) using the question concepts and the resume skill concepts as positive seeds. We then curated this knowledge graph manually to correct any errors.
The next task is assigning difficulty levels to questions. This problem has also received very little attention (Padó, 2017). We use the following simple approach. Use any automatic answer extraction technique to extract an answer text  $A$  for  $q$ , from a suitable corpus like Wikipedia or a textbook. Let  $\lambda(A)$  be the set of concepts associated with  $A$ . The degree  $d(u)$  of a concept vertex  $u$  in the knowledge graph, ignoring edge directions, is a good approximation of the "complexity" of that concept; a concept is complex, if it is directly related to many other concepts. Thus the sum of the complexities of the individual concepts in the answer to a question is a good measure of the complexity of that question:  $\gamma(q) = \sum_{u \in \lambda(q)} d(u)$ . We can now write simple rules to assign a difficulty level to each question: if  $\gamma(q) \leq c_0$  then  $\delta(q) = 0$  else if  $c_0 < \gamma(q) \leq c_1$  then  $\delta(q) = 1$  else  $\delta(q) = 2$  ( $c_0, c_1, c_2$  are user-specified constants). More complex approaches, such as applying machine learning to predict difficulty levels for questions, are possible.
Finally, we identify similar questions for our evaluation. The is_qsim() function (Eqn.3) uses overlap between annotated concepts for simplicity. Clearly, there is a need for more sophisticated approaches, for example using paraphrase detection (Galbraith et al., 2017).
To compare against out integer programming approach (IP), we use the following baselines for selecting questions for a candidate having resume  $R$ :
- BR1: Select  $n_q$  questions randomly from  $QB_s$ , where  $n_q$  is the same as the number of questions in the optimal plan.
- BR2: Let  $F_{R}(\mathfrak{s})$  denote the set of concepts related to skill  $\mathfrak{s}$  mentioned in resume  $R$ . Select  $n_{q}$  questions randomly from  $QB_{\mathfrak{s}}$ , where coverage of each selected question  $q$  has at
least one concept common with the neighbourhood of the concept set  $F_{R}(\mathfrak{s})$  i.e.,  $\psi (q)\cap \Phi (F_R(\mathfrak{s}))\neq \emptyset$
- BR3: same as BR2, but ensures even distribution of question difficulty levels.
- DuerQuiz (Qin et al., 2019): discussed in Section 2. Since no implementation is publicly available, we implemented our own version of this baseline. Since we do not use historical resumes or skill graph edge labels in our framework, we adapted DuerQuiz in the best possible way for out setting. We ignore the terms corresponding to historical resumes in the weight assignment to concepts. We further approximate descendants of concepts as their direct neighbors in the skill graph, both for weight initialization and weight propagation.
For our MIP formulation, we use the following weights and hyper-parameters:  $w_{1} = 100$ ,  $w_{2} = 100$ ,  $w_{3} = 100$ ,  $w_{4} = 30$ ,  $w_{4}^{d} = 70$ ,  $w_{5} = 30$ ,  $w_{5}^{d} = 70$ ,  $m_{0} = 0.3$ ,  $m_{1} = 0.4$ ,  $m_{2} = 0.3$ ,  $h_{0} = 0.9$  and  $T = 45$ . These weights were not finetuned, aside for  $T$  for controlling the number of recommended question. For DuerQuiz, the paper does not recommend any thumb rule for setting hyper-parameters. We set the propagation weight  $\alpha_{c} = 0.85$  by hand-tuning on one resume, and the smoothing weight  $\beta_{f} = 0.001$ .
In this section, we describe our dataset derived from real interviews, and the experiments that we conducted using this dataset. We report comparisons of our proposed approach (IP) against the baselines defined in Section 6 using both actual interview questions and as well as an user-study for evaluation. We leave out BR2 and instead use its stronger version BR3.
All experiments were performed on an Ubuntu 18.04.5 LTS machine with 8-core Intel i7-8550U 1.80GHz processors, and 16 GB memory. For IP, generation of questions with 45min time budget ( $\sim$ 25 questions) takes 155 secs on average.
Dataset: We constructed a knowledge graph of 714 concepts and 903 edges (avg. degree 2.51) from Machine Learning and Deep Learning. Our question bank consists of 549 questions from these two skills. Each question is annotated with concepts from the knowledge graph (1.18 concepts
per question on average). Finally, we use real resumes of 40 candidates (mostly fresh IT graduates) interviewed by our organization over the last year. We identify knowledge graph concepts associated with their resumes (4.7 concepts per resume on average). For 20 of these candidates, we also have the actual questions asked to them during their interviews. Of these, we consider only the questions related to our two topics of interest. The average number of questions per candidate is 5.05.
Intrinsic Evaluation on Real Interviews: In our first evaluation, we compared the set of suggested questions with the set of actually asked questions. Fig. 5 shows a comparison of our optimization formulation with the three baselines, using the forward and backward quality measures (Section 4).
As seen, our approach is clearly better than all three baselines in both evaluations and for different values of  $k_{0}$ . The differences are large for backward evaluation. The improvement against BR1 shows the importance of focusing on the resume, rather than randomly selecting questions related to the skill. The improvement against BR3 shows that just focusing on questions related to the resume is not enough. Finally, the improvement against DuerQuiz, which combines aspects of both BR1 and BR3, shows the importance of the additional terms in our objective function. Also, our analysis shows that DuerQuiz is poor at balancing between high-degree and low-degree concepts in the knowledge graph. Depending on the value of its dissipation hyper-parameter  $(\alpha_{c})$ , it either transfers all the weight of high-degree concepts to their neighbors, or does not transfer any weight from low-degree concepts to their neighbors. IP's trade-off using different terms and their corresponding weights works much better.
We further note that BR1 and BR3 perform better than DuerQuiz in terms of forward evaluation, which indicates that these generate fewer irrelevant questions. On the other hand, DuerQuiz is better than these baselines in terms of backward evaluation. This indicates that the questions generated by these baselines are more heterogeneous and lack diversity when compared against DuerQuiz to cover all questions asked during a real interview. Note that IP outperforms DuerQuiz in both directions.
Human Evaluation: Our second evaluation
Figure 5: Backward and forward intrinsic evaluations.
compares question sets generated using pairs of algorithms by 3 experienced human interviewers  $E_{1}, E_{2}, E_{3}$ . We have 3 pairs of algorithms to compare: (IP, BR1), (IP, BR3), (IP, DuerQuiz). Note that to reduce the load on the human evaluators, we did not invest in comparing the baselines with each other. We randomly assign one of these pairs to each of the  $N = 20$  candidates; e.g., 7 candidates got the pair (LP, BR1) and so forth. For each candidate, we generated two sets of questions, for the skill Machine Learning, using the algorithm pair assigned to it. Hiding the algorithm used to generate the question sets, we presented the two sets for the 20 candidates to each of the 3 experts, along with skills extracted from their resumes. For each candidate, each human expert gave a comparative ranking, indicating whether set 1 was better than set 2. We had not suggested any criterion for this comparison; each expert used her own intuition.
There were  $7 \times 3 = 21$  evaluations of (IP, BR1) pair, out of which IP "won" in 19. Using  $\chi^2$  test with  $99.9\%$  confidence, we reject the null hypothesis and accept that IP is better than BR1. Similarly, IP is better than BR3 in 14 out of 21 evaluations  $(\chi^2 85\%$  confidence). Unfortunately, IP is better than DuerQuiz in only 6 out of 21 evaluations. However, there was large disagreement among the experts in this case, and discussions showed that the experts' evaluation criteria were considerably simpler than the objective functions used in IP. For example, no expert considered the inter-linking of the questions in her evaluation, nor did they consider duplication of questions across different candidates as undesirable; but these are important factors in IP for choosing questions. In the future, we
intend to perform a larger expert study with a more nuanced evaluation which compares specific quality aspects of the question sets.
8 Conclusions and Further Work
We have proposed an interview assistant system to automatically select an optimal set of technical questions (from a question bank) personalized for a candidate. We formalized the problem of selecting a set of questions from question banks as an integer programming problem, with multiple terms in the objective functions and multiple constraints. We used knowledge graph as background knowledge, and used the candidate's resume to personalize the selection of questions. We proposed a novel intrinsic evaluation to compare a set of suggested questions with actually asked questions in real interviews. We also used expert human interviewers to comparatively evaluate our approach with a set of reasonable baselines. Our comparisons against state-of-the-art and ablated baselines show the usefulness of our proposed approach.
H.H. Chang and Z. Ying. 1996. A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20:213-229.
W.J. Van der Linden. 2005. A comparison of itemselection methods for adaptive tests with content constraints. 42:283-302.
W.J. Van der Linden and E. Boekkooi-Timminga. 1989. A maximin model for IRT-based test design with practical constraints. Psychometrika, 54:237-248.
Evanthia Faliagka, Kostas Ramantas, Athanasios Tsakalidis, and Giannis Tzimas. 2012. Application of machine learning algorithms to an online recruitment system. In Seventh International Conference on Internet and Web Applications and Services (ICIW 2012), pages 215-220.
S. Faralli, I. Finocchi, S. Ponzetto, and P. Velardi. 2018. Efficient pruning of large knowledge graphs. In *IJ-CAI*.
P. Ferragina and U. Scaiella. 2012. Fast and accurate annotation of short texts with wikipedia pages. IEEE Software, 29(01):70-75.
Byron Galbraith, Bhanu Pratap, and Daniel Shank. 2017. Talla at SemEval-2017 task 3: Identifying similar questions through paraphrase detection. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).
O. Ganea and T. Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In EMNLP.
K.T. Han. 2012. An efficiency balanced information criterion for item selection in computerized adaptive testing. Journal of Educational Measurement, 49:225-246.
Kyung (Chris) Tyek Han. 2018. Components of the item selection algorithm in computerized adaptive testing. Journal of Educational Evaluation for Health Professions, 15(7):1-13.
F. Javed, Q. Luo, M. McNair, F. Jacob, M. Zhao, and T. S. Kang. 2015. Carotene: A job title classification system for the online recruitment domain. In IEEE First International Conference on Big Data Computing Service and Applications, pages 286-293.
X. Luo. 2020. Automated test assembly with mixed-integer programming: The effects of modeling approaches and solvers. Journal of Educational Measurement.
Ulrike Padó. 2017. Question difficulty - how to estimate without norming, how to use for automated grading. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications.
G.K. Palshikar, R. Srivastava, and S. Pawar. 2017. Analytics-led talent acquisition for improving efficiency and effectiveness. In 5th IIMA International Conference on Advanced Data Analysis, Business Analytics and Intelligence (ICADABAI 2017).
S. Pawar, P. Bhattacharyya, and G.K. Palshikar. 2017. End-to-end relation extraction using neural networks and markov logic networks. In EACL.
Chuan Qin, Hengshu Zhu, Chen Zhu, Tong Xu, Fuhzen Zuang, Chao Ma, Jingshuai Zhang, and Hui Xiong. 2019. Duerquiz: A personalized question recommender system for intelligent job interview. In Proc. 25th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD'19), pages 2165-2173.
A. Sakor, K. Singh, A. Patel, and M. Vidal. 2020. Falcon 2.0: An entity and relation linking tool over wikidata. In CIKM, pages 3141-3148.
M.L. Stocking and C. Lewis. 2000. Methods of controlling the exposure of items in CAT. In Computerized adaptive testing: theory and practice, pages 163-182. Kluwer Academic Publishers.
M.L. Stocking and L. Swanson. 1993. A method for severely constrained item selection in adaptive testing. Applied Psychological Measurement, 17:277-292.
L. Swanson and M.L. Stocking. 1993. A model and heuristic for solving very large item selection. Applied Psychological Measurement, 17:151-166.
T.J. Theunissen. 1986. Some applications of optimization algorithms in test design and adaptive testing. Applied Psychological Measurement, 10:381-389.
W. J Van der Linden and C.A.W. (eds) Glas. 2000. Computerized adaptive testing: Theory and practice. Kluwer Academic Publishers.
D. Weiss. 1982. Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6:473-492.