PyNDA: Deep Learning for Psychometric Natural Language Processing

SYCHOMETRICS is concerned with the measurement of knowledge, ability, attitudes, and personality traits. With the increased importance of predictive analytics at the micro-level [1], including prediction of individuals’ behaviors [2], accurate and timely measurement of psychometrics has become of paramount importance. In health settings, psychometric measures, including health numeracy, subjective literacy, and perceptions of trust and anxiety related to physicians, have been shown to have a profound impact on various health and wellness outcomes such as future doctors’ visits and all-around well-being [6], [7], [8]. Hence, accurately and efficiently measuring psychometrics can have positive implications for many behavior prediction tasks [42], [43], [48], [49], [51], [52]. Psychometric data collection efforts have traditionally relied on survey-based methods administered on a monthly or quarterly basis. Effectively collecting and measuring relevant constructs in a timely and unobtrusive manner has proven elusive in real-world settings [9]. In recent years, machine-learning methods for natural language processing (NLP) have been successfully applied to certain psychometric dimensions such as sentiment and emotion [10]. Such NLP techniques, which analyze user-generated text and automatically score them along the target variable, afford opportunities for real-time, passive monitoring and measurement. However, several gaps and challenges remain:  Many rich psychometric dimensions remain underexplored: Whereas numerous NLP methods have been proposed for sentiment and emotion, other aspects such as attitudes, perceptions, and characteristics have received limited attention. It is unclear how effectively NLP methods can tackle these novel dimensions [50].  User-centric versus task-centric modeling: Most prior NLP classification objectives and data sets have been arranged around a given task (e.g., sentiment polarity). Psychometric dimensions such as attitudes and perceptions are very individualized, with multiple interrelated target variables of interest associated with each person. There is an opportunity for psychometric NLP methods to incorporate provisions for user-centric modeling [44], [47].  Demographic-sensitive modeling: Factors such as age, race, gender, and education can have a profound impact on various psychometric measures (e.g., literacy, trust, anxiety) [5]. These differences can be amplified in usergenerated text [11]. Several recent studies suggest that machine learning models that fail to properly control for demographics are prone to inaccurate generalizations [12], [46]. Psychometric NLP methods that are accurate across diverse demographic populations are a necessary and understudied research area [45].  Paucity of available text: Several recent NLP studies have examined “short-text” contexts such as Twitter [13] and news articles [14]. User-generated text associated with psychometrics often appears in similarly sparse environments such as comment boxes, text messages, and microblogs, necessitating methods capable of learning patterns from limited linguistic cues.