SlowGuess/NAACL_2021_OCR huggingface dataset:
Soham Datta, Prabir Mallick, Sangameshwar Patil
d.soham, mallick.prabir, sangameshwar.patil
Indrajit Bhattacharya, Girish Palshikar
b.indrajit, gk.palshikar, @tcs.com
TCS Research
Given the diversity of the candidates and complexity of job requirements, and since interviewing is an inherently subjective process, it is an important task to ensure consistent, uniform, efficient and objective interviews that result in high quality recruitment. We propose an interview assistant system to automatically, and in an objective manner, select an optimal set of technical questions (from question banks) personalized for a candidate. This set can help a human interviewer to plan for an upcoming interview of that candidate. We formalize the problem of selecting a set of questions as an integer linear programming problem and use standard solvers to get a solution. We use knowledge graph as background knowledge in this formulation, and derive our objective functions and constraints from it. We use candidate's resume to personalize the selection of questions. We propose an intrinsic evaluation to compare a set of suggested questions with actually asked questions. We also use expert interviewers to comparatively evaluate our approach with a set of reasonable baselines.
A large multi-national IT company added roughly 70,000 employees in FY2018-19. Assuming an average interview time of 30 minutes, 3 interviewers in each interview, and 4 candidates interviewed for every position, implies approximately 420,000 person-hours were spent in one year just on conducting the interviews. Given the diversity of the candidates and complexity of job requirements, and considering that interviewing is an inherently human and subjective process, it is a mammoth task to ensure consistent, uniform, efficient and objective interviews that result in high quality re
cruitment. AI and ML technologies are increasingly playing important roles in helping improve recruitment quality, e.g., Faliagka et al. (2012), Javed et al. (2015), Palshikar et al. (2017), although ethical issues are emerging. In this paper, we consider one particular way to assist human interviewers in improving the quality of their interviews and in reducing subjectivity.
Before conducting an interview, an interviewer typically studies the candidate's resume, noting salient points about her education, skills, job history, roles, projects, tasks handled etc. The interviewer also notes the apparent strengths and weaknesses of the candidate, as also the extent to which she matches (and does not match) the job profile for which she will be interviewed. In short, the interviewer builds an a priori rough mental profile of the candidate, and prepares a mental plan of how to interview her. Such a plan includes preparing an unordered set of questions that the interviewer would like to ask the candidate. In this paper, we propose an interview assistant system to automatically, and in an objective, unbiased manner, build such a set of questions for a human interviewer, which can be part of a plan for an upcoming interview. We assume we have question banks from where questions can be selected.
Note that such a plan is static, and the actual sequence of questions asked by an interviewer may diverge from the static plan, due to dynamic and contextual reasons observed during the flow of the interview. Such reasons include (i) mismatch between the interviewer's prior impression about the strengths of the candidate and the quality of the candidate's actual answers; (ii) the questions asked by other interviewers, if they are present. Nevertheless, such a plan generated by the system is still useful, as it reduces the cognitive load on the interviewer, and brings some standardization
<sup>1</sup>https://www.tcs.com/content/dam/tcs/investor-relations/financial-statements/2018-19/ar/annual-report-2018-2019.pdf
$^{2}$ https://hbr.org/2019/04/the-legal-and-ethical-implications-of-using-ai-in-hiring
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1996-2005 June 6-11, 2021. ©2021 Association for Computational Linguistics
and objectivity to an interview. Having a system-suggested set of questions, personalized for a particular candidate, before starting the interview is useful for the interviewer. The questions help in getting to a good start and also give ideas about where to focus during the interview.
The novel contributions of this paper are as follows. We formalize the problem of selecting a set of questions as an integer programming optimization problem and use standard solvers to get a solution. We use knowledge graph as background knowledge, and formulate our objective functions and constraints from it. To our knowledge, this is the first paper to address the problem of creating an optimal interview plan. We report experiments on a real dataset of candidates and interview questions and compare against a state-of-the-art baseline using both intrinsic and human study based evaluation.
The rest of the paper is organized as follows. Section 2 summarizes related work, Section 3 formulates the optimization problem, Section 4 gives details of our novel evaluation measure, Section refsec: nlp describes the use of NLP techniques to build the requisite resources, Section 6 describes the baselines used for comparison, Section 7 describes the our experimental results, and Section 8 concludes the paper.
Work in this paper is close to the field of computerized adaptive testing (CAT) (Van der Linden and Glas, 2000), where the task is to select questions (also called items) on-the-fly from a question bank, depending on how the student has answered the questions so far (i.e., adjusting to her ability level), with goals of creating shorter tests that yield better differentiation among students. CAT techniques are used in large-scale general online examinations like GRE, and in specialized medical licensing or certification examinations, e.g., in clinical pathology, emergency medicine, and pharmacy. We are not aware of any work on applying CAT techniques to interviews.
We first outline the key differences between our work and CAT, which stem from the obvious differences between interviews and examinations. Interviews are not really examinations, but are human, face-to-face, oral, and two-way interactions among a single candidate and possibly multiple interviewers. There is no set question paper,
no rigid time-limit and interview questions need short free-form textual or spoken answers whereas CAT deals mostly with examinations administering multiple-choice questions. Unlike an examination, the interactions are two-way; e.g., the candidate can ask for clarification about a question. Goals of an interview are different from that of an examination, e.g., assessing fitment for job position requiring multiple skills, rather than assessing depth and breadth in a fixed subject. Students are anonymous in CAT, whereas interviewers have detailed knowledge about the candidate; e.g., through her resume. CAT is about a dynamically assembled, personalized, sequence of questions which are dependent on the student's answers so far, whereas in this paper we deal with a static one-time selection of interview questions, with no dynamic adjustment as per the candidate's answers i.e., in this paper we cannot estimate the candidate's latent abilities, since we do not have her answers. This prevents a direct comparison of our work with most CAT techniques.
See Han (2018) for a review of CAT research. The key aspects of CAT are: item selection criteria, content balancing (ensuring coverage of all sub-areas in the subject) and item exposure control (using randomization to prevent excessive item reuse across multiple examinations). Many item selection approaches are formulated using item response theory and use information-theoretic criteria; e.g., Fisher information (Weiss, 1982), efficiency balanced information (EBI) (Han, 2012), Kullback-Liebler information (Chang and Ying, 1996). Various item exposure control methods have been proposed to reduce overuse of "good" items; see Stocking and Lewis (2000) for a survey of early methods.
While some CAT systems use the above 3 aspects separately, the automated test assembly (ATA) approaches use them together in an optimization framework such as linear programming or mixed integer programming, where content balancing criteria are constraints and item selection criteria are objective functions; e.g., Theunissen (1986), der Linden and Boekkooi-Timminga (1989), Stocking and Swanson (1993), Swanson and Stocking (1993). For a comparative analysis of such optimization approaches, see der Linden (2005) and Luo (2020). Again, it is difficult to directly compare our work with these optimization-based approaches, because of our static setting
in interviews where no answers are available to estimate candidate proficiency (unlike CAT). In most CAT approaches, the information available for each item is rather limited: subject, sub-area, difficulty level, discrimination level etc. We have used knowledge graphs to create a semantically rich and detailed characterization of questions in terms of concepts. Our optimization formulation uses the knowledge graph to generate novel constraints (including content balancing) and objective functions for item selection.
CAT algorithms cannot be directly used as baselines, because (i) they output an ordered sequence of questions (we output an unordered set of questions); and (ii) they need candidate answers, which are not available to us.
DuerQuiz (Qin et al., 2019) starts from job descriptions and resumes of candidates, and considers the problem of recommending a set of questions using a skill graph. It uses knowledge of resumes of candidates who have been hired in the past. It additionally considers the task of extracting skills from resumes and job descriptions, and construction of the skill graph, which are not our primary focus. For the actual task of question selection for a specific resume, DuerQuiz initializes weights of concepts based on the job description, historical resumes and the focus resume, and then dissipates those weights over descendant concepts in the skill graph. Finally, the weights determine the number of questions selected from a concept. It does not consider the notion of question difficulty, or relations between questions.
For concreteness, in this paper we focus on candidates in the IT domain. We start by noting that an interviewer asks different types of questions. Technical questions explore the breadth and depth of the candidate's understanding of a particular technical skill. Other than technical questions, interviewers also ask techno-experience questions (e.g. about skills in projects), methodological questions, behavioural questions, among others. For concreteness, in this paper we focus only on technical questions about skills in the IT domain. We focus on entry-level candidates (freshers or those with less than 1 year experience), because for more experienced candidates the interviewers tend to move quickly to techno-experience questions. We also assume the ques
$$
\text {m a x i m i z e} f _ {1}: \sum_ {i = 1} ^ {| Q |} x _ {i} \cdot | \psi (q _ {i}) | +
$$
$$
f _ {2}: \sum_ {i = 1} ^ {| Q |} \sum_ {j > i} ^ {| Q |} x _ {i} \cdot x _ {j} \cdot i s _ {-} q g r a p h _ {-} e d g e (q _ {i}, q _ {j}) +
$$
$$
f _ {3}: \sum_ {i = 1} ^ {| Q |} \sum_ {j > i} ^ {| Q |} x _ {i} \cdot x _ {j} \cdot \neg i s \_ q g r a p h \_ p a t h (q _ {i}, q _ {j}) +
$$
$$
f _ {4}: \sum_ {i = 1} ^ {| Q |} x _ {i} \cdot (\psi (q _ {i}) \cap \Phi (W _ {R}) \neq \emptyset) +
$$
$$
f _ {5}: \sum_ {i = 1} ^ {| Q |} x _ {i} \cdot (\psi (q _ {i}) \cap \Phi (W _ {J}) \neq \emptyset)
$$
$$
\text {s u c h t h a t} \quad \mathrm {C} 1: \sum_ {i = 1} ^ {| Q |} x _ {i} \cdot T _ {\delta \left(q _ {i}\right)} \left(q _ {i}\right) \leq T
$$
$$
\mathrm {C} 2 (\mathrm {k}): \sum_ {i = 1} ^ {| Q |} x _ {i} \cdot (\delta (q _ {i}) = = k) \leq \left(m _ {k} \cdot \left(\sum_ {i = 1} ^ {| Q |} x _ {i}\right)\right)
$$
$$
\mathrm {C} 5: \sum_ {i = 1} ^ {| Q |} x _ {i} \cdot \delta (q _ {i}) \geq h _ {0} \cdot \sum_ {i = 1} ^ {| Q |} x _ {i}
$$
Figure 1: The integer programming problem
tions are such that they require short answers, typically containing up to 5-6 sentences.
Given a candidate resume  $R$ , a technical skill  $\mathsf{s}$  and a question bank  $QB_{\mathsf{s}}$  about that skill, the task we address is: how to select the "best" questions from  $QB_{\mathsf{s}}$ , which maximize some objective functions and meet the required constraints? The questions need to be selected from  $QB_{\mathsf{s}}$  and need to be highly personalized for the candidate in the sense that they should be closely related to the candidate's background mentioned in  $R$ . The complete optimization formulation is given in Fig. 1.
We use the term skill to refer to a broad technical area; examples: Python, Machine_Learning, Algorithms, Networking etc. Let s denote a given skill. Let  $C_{\mathbf{s}}$  (or just  $C$  if the skill is clear) be a set of concepts related to a given skill. Relationships such as IS-A, HAS-A (inverse of IS-PART-OF) hold between pairs of concepts; e.g., array IS-A data_structure and class HAS-A method. We represent the concepts in a particular skill and their inter-relationships as a knowledge graph  $G = (C,E,\eta)$ , where the vertices are concepts,  $E$  is the set of directed edges linking pairs of con
cepts, and  $\eta : E \to REL$  is the edge labeling function that associates a relationship name with every edge. Fig. 3 shows a small part of a concept graph for the skill Python; here, each vertex corresponds to a concept, the edges show relationships between concepts and the edge label is shown on each arrow.
Neighbourhood of a concept  $u \in C$ , denoted  $\phi(u)$ , is the set of concepts directly connected to  $u$  in the knowledge graph, along with  $u$  itself. For simplicity, we ignore the edge direction and edge label when computing  $\phi(u)$ . Example:  $\phi(11) = \{4, 11, 12, 15\}$ . Neighbourhood of a set of concepts  $B = \{u_1, \ldots, u_k\}$  is the union of their neighbourhoods:  $\Phi(B) = \phi(u_1) \cup \ldots \cup \phi(u_k)$ .
We assume we have a question bank, which is a set of questions about a particular skill, along with some other information with each question. Formally, a question bank for a skill  $\mathsf{s}$  is  $QB_{\mathsf{s}} = (Q,\delta ,\lambda)$ , where  $Q$  is a set of questions about  $\mathsf{s}$ , the function  $\delta (q)$  associates a difficulty level with every question  $q\in Q$ , and the function  $\lambda (q)$  associates a non-empty subset of concepts with every question  $q\in Q$ . A difficulty level is 0 (easy), 1 (medium), 2 (hard); a more nuanced scale can be easily incorporated. Fig. 2 shows a small question bank containing 13 questions for the skill Python; it also shows the difficulty level and the subset of concepts (from the knowledge graph of Fig. 3) associated with each question. In order to prevent the same subset of questions being identified by our solution for very similar resumes, we could either shuffle the questions in  $Q$ , or use only a random subset of  $Q$  as input.
Coverage of a question  $q$ , denoted  $\psi(q)$ , is the set of concepts  $\lambda(q)$  associated with  $q$ , along with the concepts at 1-hop from each concept in  $\lambda(q)$ . For simplicity, we ignore the edge direction and edge label when computing  $\psi(q)$ . Example:  $\psi(q_2) = \{20, 21, 17, 24\}$ . Let  $x_i, 1 \leq i \leq |Q|$  be the set of Boolean variables, where if  $x_i = 1$  the question  $q_i$  is included in a set of questions, and not included if  $x_i = 0$ . Then the first term  $f_1$  in our objective function selects a set of questions which has the maximum coverage.
Different candidates can take different amounts of time  $T(q)$  for answering any particular question  $q$  in the QB. This time is candidate specific and unknown a priori. We have a simple model
to accommodate this time: a candidate takes time  $T_{i}(q)$  minutes to answer a question  $q$  having difficulty level  $\delta(q) = i$ ; for concreteness, we assume  $T_{0}(q) = 1, T_{1}(q) = 2, T_{2}(q) = 3$ . This simplified model predicts the same time, for all candidates, for all questions having a particular difficulty level; a more nuanced approach would be, for example, to learn the time distribution from data of past interviews. Interviewers often have an informal time-limit (budget)  $T$  on the time to spend on a particular skill. So we have constraint C1: the total estimated time taken to answer the selected questions must be at most  $T$ .
In order to prevent selection of only easy questions, we introduce constraints  $\{\mathrm{C2(j)}\}$  for  $j\in 0,1,2$  that force a more equitable user-specified distribution of difficulty levels in selected questions. The user-specified constants  $0\leq m_0,m_1,m_2\leq 1,m_0 + m_1 + m_2 = 1$  give control to the user to generate questions that "suit" a particular "style"; e.g., setting  $m_0 = 0.2,m_1 = 0.2,m_2 = 0.6$  will tend to select more hard questions.
Questions asked in an interview are often related to another question, indicating exploration of the depth of a candidate's knowledge. Given a set of questions  $A = \{q_1, \ldots, q_k\}$ , we define a question graph  $G_A$ , whose vertices are the questions in  $A$  and two questions  $q_i, q_j$  have an undirected edge if  $\lambda(q_i) \cap \lambda(q_j) \neq \emptyset$ . In general,  $G_A$  may be a disconnected graph. A path of length 1 or more indicates a sequence of inter-related questions. Fig. 4 shows the question graph for the questions in Fig. 2. A path  $P$  in a graph is a longest path (or chain) if  $P$  is not a sub-path of any other path in the graph. Now we have another term  $f_2$  in our objective function: maximize the sum of the lengths of all longest paths in  $G_A$ . Since this is computationally expensive, we can use an approximation the number of edges in  $G_A$ , since an edge indicates a sequence of two questions. Note that this term has a quadratic form, which can be easily linearized taking advantage of the fact that the decision variables are all binary (we omit this reformulation).
Questions asked in an interview are often unrelated to another question, indicating exploration of the breadth of a candidate's knowledge. We define two questions  $q_{i}, q_{j}$  as unrelated if there is no path between them in  $G_{A}$  i.e.,  $q_{i}$  is unreachable from  $q_{j}$  and vice versa. Analogously, we define
3A more realistic setting would associate an ordered sequence of concepts with a question, ranked in terms of the decreasing relevance of the concepts to the question.
two paths  $P_{1}, P_{2}$  in  $G_{A}$  as unrelated if no vertex in  $P_{2}$  is reachable from any vertex in  $P_{1}$  and vice versa. Now we have another term  $f_{3}$ : maximize the number of pairs of paths which are unrelated. Since this is computationally expensive, we can use as an approximation the number of pairs of vertices which are unreachable via paths from each other. Note that this objective also has the quadratic form, which we linearize.
An interview often includes questions about concepts related to a given skill which are mentioned in the candidate's resume  $R$ . Let  $W_{R} = \{w_{1}, w_{2}, \ldots, w_{\ell}\}$  denote the  $\ell$  concepts mentioned in the candidate's resume; e.g., in the descriptions of her jobs, projects, trainings etc. A reasonable interview must include questions related to as many concepts in the neighbourhood of  $W_{R}$  as possible, giving us another objective function term  $f_{4}$ . We can further refine this objective to specifically consider questions directly related to  $W_{R}$ , giving us an additional term  $f_{4}^{d}$ , with  $W_{R}$  instead of  $\Phi(W_{R})$  and  $\lambda(q)$  instead of  $\psi(q)$ . We could refine this even further, if we could estimate from the resume that the candidate has different proficiency levels in different concepts; e.g., if a candidate has worked in Flash in 1 project of 3 months and in numpy in two projects for 11 months, then she is clearly stronger in numpy than in Flash.
Analogously, let  $W_{J}$  denote the set of concepts relevant to a given job description for which the candidate is being interviewed. A reasonable interview must include questions related to as many concepts in the neighborhood  $W_{J}$  as possible, giving us term  $f_{5}$ . As for  $f_{4}$ , here as well we consider a direct version  $f_{5}^{d}$ , with  $W_{J}$  replacing  $\Phi(W_{J})$  and  $\lambda(q)$  replacing  $\psi(q)$ .
Suppose we have some idea of the proficiency level  $\eta (\mathsf{s})$  that a particular candidate has in a given skill s. This estimate could be generated from the information in the resume (projects, tasks, trainings) or from other sources, such as the scores in a prior written test. Suppose the estimated proficiency level in a skill is an integer from 0 (does not know) to 4 (expert). We should take this input into account in order to adjust the difficulty level of selected questions; e.g., a candidate with proficiency level  $\eta (\mathsf{s}) = 3$  should be asked fairly difficult questions. This gives us constraint C5, which says that the average difficulty level of selected questions should be above a user-specified
1. What is the difference between list and tuple in Python? 0 {20 22}
2. What is the difference between an array and a list in Python? 1{20 21}
3. What is a dictionary in Python? 1{23}
4. How do you implement a 2D array in Python? 1 {24}
5. How are exceptions handled in Python code? 1{12 16}
6. How does Python support encapsulation? 0 {2}
7. What facilities does Python have for data abstraction? 1 {3}
8. What is __init__ in Python? 0 {15 12}
9. What is a module in Python? 0 {9}
10. What is pickling and unpickling in Python? 1 {18}
11. How does one share global variables across modules in a Python program? 2 {7 9 12}
12. Explain what is Flask and what are its benefits. 2 {10}
13. How do you use Flask to create a database table? 2 {10 13 14}
Figure 2: A small question bank for the skill Python.
Figure 3: A knowledge (sub)graph for skill Python.
constant  $h_0$ , which can be derived from the proficiency level  $\eta(\mathsf{s})$  of the candidate in skill  $\mathsf{s}$ .
We normalize the terms in the objective function so that these take values in  $[0,1]$ . Further, we take a weighted sum (instead of the plain sum) of the terms:  $w_{1} \cdot f_{1} + \ldots + w_{5} \cdot f_{5}$ , where  $w_{1}, \ldots, w_{4}, w_{4}^{d}, w_{5}, w_{5}^{d}$  are user-given positive real weights. The weights will allow the interviewer to change the relative importance of the terms.
Interview Plans for a Set of Candidates: The optimization program discussed so far is useful to generate an interview plan for one particular candidate. However, in many situations (such as campus interviews), there is a sequence of interviews for multiple candidates and the system is required to generate a system plan for each of them. There are additional constraints on the set of interview plans generated in such a situation. For example, the repetition of questions across multiple candidates should be minimized i.e., different candidates (even those having a similar background) should by-and-large get different sets of questions.
Let  $N_0$  denote the number of candidates to be interviewed in the current campaign. We assume that a single question bank  $QB_{s}$  for skill  $s$  (or, just  $Q$  for simplicity) will be used to se
Figure 4: Question graph for the example questions.