DETERMINING INTERPERSONAL OR BEHAVIORAL SKILLS BASED ON COURSE INFORMATION

Embodiments are directed to determining interpersonal or behavioral skills based on course information in course offerings. One or more learning objectives may be determined from course information associated with a course such that each learning objective may be associated with a learning objective narrative. A skills model may be employed to determine one or more soft skills based on the one or more learning objective narratives. The one or more soft skills may be associated with a course profile that corresponds to the course. In response to a quality score associated with the skills model being less than a threshold value, the skills model may be retrained with one or more reference models having one or more portions based on machine learning.

TECHNICAL FIELD

The present invention relates generally to data management, and more particularly, but not exclusively, to managing data for determining interpersonal or behavioral skills based on course information.

BACKGROUND

Identifying persons that make good employees has long been a goal of organizations. And, in today's highly competitive global market, finding and keeping great employees is becoming more challenging. Conventionally, organizations may be forced to rely on narrow or limited criteria derived from anecdotal evidence, personal preferences, gut feelings, or the like, rather than evidence based analytics to determine if a person may be a good employee candidate. Similarly, educational institutions may want to provide educational opportunities that provide their students desirable employment opportunities. Accordingly, in some cases, educational institutions may design offerings based on their perception of the needs of desirable employers. But, not unlike employers, educational institutions may have limited access to evidence based analytics to help them design their offerings. Further, students may seek out educational institutions that to prepare them for careers with desirable employers or careers. Accordingly, educational institutions often provide course information that students may employ to evaluate if a particular course may be applicable to their career goals. Likewise, employers may review course information for courses taken by candidate employees to evaluate how well candidate employees may fit the employers' needs. Course information may describe some of the activities or skills taught by a course. However, in some cases, discerning a complete view of the skills learned from a course may be difficult for employers or students alike. Thus, it is with respect to these considerations and others that the present invention has been made.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

As used herein the term, “engine” refers to logic embodied in hardware or software instructions, which can be written in a programming language, such as C, C++, Objective-C, COBOL, Java™, PHP, Perl, Python, JavaScript, Ruby, VBScript, Microsoft .NET™ languages such as C #, or the like. An engine may be compiled into executable programs or written in interpreted programming languages. Software engines may be callable from other engines or from themselves. Engines described herein refer to one or more logical modules that can be merged with other engines or applications or can be divided into sub-engines. The engines can be stored in non-transitory computer-readable medium or computer storage device and be stored on and executed by one or more general purpose computers, thus creating a special purpose computer configured to provide the engine.

As used herein the term “skills model” refers one or more data structures that encapsulate the data, rules, machine learning models, machine learning classifiers, heuristics, or instructions that may be employed to determine soft skills from course information. Skills models may include various components, such as, one or more machine learning based classifiers, heuristics, rules, pattern matching, conditions, or the like. Skills models may be provided learning objective narratives to classify if the provided learning objectives may be associated with one or more soft skills.

As used herein the term “course profile” refers to one or more data structures or records gathered together to provide information about a course. For example, a course profile may include (or reference) various course information, candidate skills, confirmed skills, scorer history, or the like. Interpersonal or behavioral skills (e.g., soft skills) associated with a course may be included in a course profile.

As used herein the term “learner profile” refers to one or more data structures or records gathered together to provide information about a student. For example, a learner profile may include (or reference) various information, such as, soft skills, hard skills, educational history, employment history, short term learning/employment goals, long term learning/employment goals, demographic data, or the like. For example, interpersonal or behavioral skills (e.g., soft skills) associated with courses completed by a learner may be included in the learner's learner profile.

As used herein the term “configuration information” refers to information that may include rule-based policies, pattern matching, scripts (e.g., computer readable instructions), or the like, that may be provided from various sources, including, configuration files, databases, user input, built-in defaults, plug-ins, extensions, or the like, or combination thereof.

Briefly stated, various embodiments are directed to determining interpersonal or behavioral skills based on course information in course offerings. In one or more of the various embodiments, one or more learning objectives may be determined from course information associated with a course such that each learning objective may be associated with a learning objective narrative.

In one or more of the various embodiments, a skills model may be employed to determine one or more soft skills based on the one or more learning objective narratives.

In one or more of the various embodiments, the one or more soft skills may be associated with a course profile that corresponds to the course.

In one or more of the various embodiments, in response to a quality score associated with the skills model being less than a threshold value, the skills model may be retrained with one or more reference models having one or more portions based on machine learning.

In one or more of the various embodiments, determining the one or more learning objectives may include: ingesting one or more of a course syllabus or a course catalog associated with the course; providing contents of the course information to an extraction model to determine the one or more learning objections or the one or more learning objective narratives; providing the one or more learning objectives or the one or more learning objective narratives to a skills engine; or the like.

In one or more of the various embodiments, employing the skills model to determine one or more soft skills may include, providing a soft skill taxonomy associated with one or more words or one or more phrases with one or more soft skills such that soft skills include one or more of creativity, communication, collaboration, community contribution, leadership, work ethic, adaptability, problem solving, or the like.

In one or more of the various embodiments, employing one or more natural language processing actions declared in the skills model to determine the one or more soft skills based on matching the one or more learning objective narratives with a soft skill taxonomy.

In one or more of the various embodiments, the skills model may be determined based on one or more characteristics of the course information such that the one or more characteristics include one or more of a course type, a course subject matter, a syllabus format, a syllabus content, or the like.

In one or more of the various embodiments, the quality score may be determined based on one or more quality metrics associated with the skills model such that the one or more quality metrics may be collected from one or more of a user-interface that provides a user satisfaction or one or more measured user interactions with the course profile.

In one or more of the various embodiments, other course information associated with one or more courses may be employed to generate one or more skills models based on one or more of machine learning, experimental observation, configurable heuristics, or the like.

Illustrated Operating Environment

FIG.1shows components of one embodiment of an environment in which embodiments of the invention may be practiced. Not all of the components may be required to practice the invention, and variations in the arrangement and type of the components may be made without departing from the spirit or scope of the invention. As shown, system100ofFIG.1includes local area networks (LANs)/wide area networks (WANs)-(network)110, wireless network108, client computers102-105, predictive learner server computer116, or the like.

At least one embodiment of client computers102-105is described in more detail below in conjunction withFIG.2. In one embodiment, at least some of client computers102-105may operate over one or more wired or wireless networks, such as networks108, or110. Generally, client computers102-105may include virtually any computer capable of communicating over a network to send and receive information, perform various online activities, offline actions, or the like. In one embodiment, one or more of client computers102-105may be configured to operate within a business or other entity to perform a variety of services for the business or other entity. For example, client computers102-105may be configured to operate as a web server, firewall, client application, media player, mobile telephone, game console, desktop computer, or the like. However, client computers102-105are not constrained to these services and may also be employed, for example, as for end-user computing in other embodiments. It should be recognized that more or less client computers (as shown inFIG.1) may be included within a system such as described herein, and embodiments are therefore not constrained by the number or type of client computers employed.

Computers that may operate as client computer102may include computers that typically connect using a wired or wireless communications medium such as personal computers, multiprocessor systems, microprocessor-based or programmable electronic devices, network PCs, or the like. In some embodiments, client computers102-105may include virtually any portable computer capable of connecting to another computer and receiving information such as, laptop computer103, mobile computer104, tablet computers105, or the like. However, portable computers are not so limited and may also include other portable computers such as cellular telephones, display pagers, radio frequency (RF) devices, infrared (IR) devices, Personal Digital Assistants (PDAs), handheld computers, wearable computers, integrated devices combining one or more of the preceding computers, or the like. As such, client computers102-105typically range widely in terms of capabilities and features. Moreover, client computers102-105may access various computing applications, including a browser, or other web-based application.

A web-enabled client computer may include a browser application that is configured to send requests and receive responses over the web. The browser application may be configured to receive and display graphics, text, multimedia, and the like, employing virtually any web-based language. In one embodiment, the browser application is enabled to employ JavaScript, HyperText Markup Language (HTML), extensible Markup Language (XML), JavaScript Object Notation (JSON), Cascading Style Sheets (CSS), or the like, or combination thereof, to display and send a message. In one embodiment, a user of the client computer may employ the browser application to perform various activities over a network (online). However, another application may also be used to perform various online activities.

Client computers102-105also may include at least one other client application that is configured to receive or send content between another computer. The client application may include a capability to send or receive content, or the like. The client application may further provide information that identifies itself, including a type, capability, name, and the like. In one embodiment, client computers102-105may uniquely identify themselves through any of a variety of mechanisms, including an Internet Protocol (IP) address, a phone number, Mobile Identification Number (MIN), an electronic serial number (ESN), a client certificate, or other device identifier. Such information may be provided in one or more network packets, or the like, sent between other client computers, predictive learner server computer116, or other computers.

Client computers102-105may further be configured to include a client application that enables an end-user to log into an end-user account that may be managed by another computer, such as predictive learner server computer116, or the like. Such an end-user account, in one non-limiting example, may be configured to enable the end-user to manage one or more online activities, including in one non-limiting example, project management, software development, system administration, configuration management, search activities, social networking activities, browse various websites, communicate with other users, or the like. Also, client computers may be arranged to enable users to display reports, interactive user-interfaces, or results provided by predictive learner server computer116, or the like.

Wireless network108is configured to couple client computers103-105and its components with network110. Wireless network108may include any of a variety of wireless sub-networks that may further overlay stand-alone ad-hoc networks, and the like, to provide an infrastructure-oriented connection for client computers103-105. Such sub-networks may include mesh networks, Wireless LAN (WLAN) networks, cellular networks, and the like. In one embodiment, the system may include more than one wireless network.

Wireless network108may further include an autonomous system of terminals, gateways, routers, and the like connected by wireless radio links, and the like. These connectors may be configured to move freely and randomly and organize themselves arbitrarily, such that the topology of wireless network108may change rapidly.

Wireless network108may further employ a plurality of access technologies including 2nd (2G), 3rd (3G), 4th (4G) 5th (5G) generation radio access for cellular systems, WLAN, Wireless Router (WR) mesh, and the like. Access technologies such as 2G, 3G, 4G, 5G, and future access networks may enable wide area coverage for mobile computers, such as client computers103-105with various degrees of mobility. In one non-limiting example, wireless network108may enable a radio connection through a radio network access such as Global System for Mobil communication (GSM), General Packet Radio Services (GPRS), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Wideband Code Division Multiple Access (WCDMA), High Speed Downlink Packet Access (HSDPA), Long Term Evolution (LTE), and the like. In essence, wireless network108may include virtually any wireless communication mechanism by which information may travel between client computers103-105and another computer, network, a cloud-based network, a cloud instance, or the like.

Network110is configured to couple network computers with other computers, including, predictive server computer116, client computers102, and client computers103-105through wireless network108, or the like. Network110is enabled to employ any form of computer readable media for communicating information from one electronic device to another. Also, network110can include the Internet in addition to local area networks (LANs), wide area networks (WANs), direct connections, such as through a universal serial bus (USB) port, Ethernet port, other forms of computer-readable media, or any combination thereof. On an interconnected set of LANs, including those based on differing architectures and protocols, a router acts as a link between LANs, enabling messages to be sent from one to another. In addition, communication links within LANs typically include twisted wire pair or coaxial cable, while communication links between networks may utilize analog telephone lines, full or fractional dedicated digital lines including T1, T2, T3, and T4, or other carrier mechanisms including, for example, E-carriers, Integrated Services Digital Networks (ISDNs), Digital Subscriber Lines (DSLs), wireless links including satellite links, or other communications links known to those skilled in the art. Moreover, communication links may further employ any of a variety of digital signaling technologies, including without limit, for example, DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12, OC-48, or the like. Furthermore, remote computers and other related electronic devices could be remotely connected to either LANs or WANs via a modem and temporary telephone link. In one embodiment, network110may be configured to transport information of an Internet Protocol (IP).

Also, one embodiment of predictive learner server computer116is described in more detail below in conjunction withFIG.3. AlthoughFIG.1illustrates predictive learner server computer116, or the like, each as a single computer, the innovations or embodiments are not so limited. For example, one or more functions of predictive learner server computer116, or the like, may be distributed across one or more distinct network computers. Moreover, in one or more embodiments, predictive learner server computer116may be implemented using a plurality of network computers. Further, in one or more of the various embodiments, predictive learner server computer116, or the like, may be implemented using one or more cloud instances in one or more cloud networks. Accordingly, these innovations and embodiments are not to be construed as being limited to a single environment, and other configurations, and other architectures are also envisaged.

Illustrative Client Computer

FIG.2shows one embodiment of client computer200that may include many more or less components than those shown. Client computer200may represent, for example, one or more embodiment of mobile computers or client computers shown inFIG.1.

Client computer200may include processor202in communication with memory204via bus228. Client computer200may also include power supply230, network interface232, audio interface256, display250, keypad252, illuminator254, video interface242, input/output interface238, haptic interface264, global positioning systems (GPS) receiver258, open air gesture interface260, temperature interface262, camera(s)240, projector246, pointing device interface266, processor-readable stationary storage device234, and processor-readable removable storage device236. Client computer200may optionally communicate with a base station (not shown), or directly with another computer. And in one embodiment, although not shown, a gyroscope may be employed within client computer200to measuring or maintaining an orientation of client computer200.

Power supply230may provide power to client computer200. A rechargeable or non-rechargeable battery may be used to provide power. The power may also be provided by an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the battery.

Audio interface256may be arranged to produce and receive audio signals such as the sound of a human voice. For example, audio interface256may be coupled to a speaker and microphone (not shown) to enable telecommunication with others or generate an audio acknowledgment for some action. A microphone in audio interface256can also be used for input to or control of client computer200, e.g., using voice recognition, detecting touch based on sound, and the like.

Display250may be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LED), Organic LED (OLED) or any other type of light reflective or light transmissive display that can be used with a computer. Display250may also include a touch interface244arranged to receive input from an object such as a stylus or a digit from a human hand, and may use resistive, capacitive, surface acoustic wave (SAW), infrared, radar, or other technologies to sense touch or gestures.

Keypad252may comprise any input device arranged to receive input from a user. For example, keypad252may include a push button numeric dial, or a keyboard. Keypad252may also include command buttons that are associated with selecting and sending images.

Illuminator254may provide a status indication or provide light. Illuminator254may remain active for specific periods of time or in response to event messages. For example, when illuminator254is active, it may back-light the buttons on keypad252and stay on while the client computer is powered. Also, illuminator254may back-light these buttons in various patterns when particular actions are performed, such as dialing another client computer. Illuminator254may also cause light sources positioned within a transparent or translucent case of the client computer to illuminate in response to actions.

Further, client computer200may also comprise hardware security module (HSM)268for providing additional tamper resistant safeguards for generating, storing or using security/cryptographic information such as, keys, digital certificates, passwords, passphrases, two-factor authentication information, or the like. In some embodiments, hardware security module may be employed to support one or more standard public key infrastructures (PKI), and may be employed to generate, manage, or store keys pairs, or the like. In some embodiments, HSM268may be a stand-alone computer, in other cases, HSM268may be arranged as a hardware card that may be added to a client computer.

Client computer200may also comprise input/output interface238for communicating with external peripheral devices or other computers such as other client computers and network computers. The peripheral devices may include an audio headset, virtual reality headsets, display screen glasses, remote speaker system, remote speaker and microphone system, and the like. Input/output interface238can utilize one or more technologies, such as Universal Serial Bus (USB), Infrared, WiFi, WiMax, Bluetooth™, and the like.

Input/output interface238may also include one or more sensors for determining geolocation information (e.g., GPS), monitoring electrical power conditions (e.g., voltage sensors, current sensors, frequency sensors, and so on), monitoring weather (e.g., thermostats, barometers, anemometers, humidity detectors, precipitation scales, or the like), or the like. Sensors may be one or more hardware sensors that collect or measure data that is external to client computer200.

Haptic interface264may be arranged to provide tactile feedback to a user of the client computer. For example, the haptic interface264may be employed to vibrate client computer200in a particular way when another user of a computer is calling. Temperature interface262may be used to provide a temperature measurement input or a temperature changing output to a user of client computer200. Open air gesture interface260may sense physical gestures of a user of client computer200, for example, by using single or stereo video cameras, radar, a gyroscopic sensor inside a computer held or worn by the user, or the like. Camera240may be used to track physical eye movements of a user of client computer200.

GPS transceiver258can determine the physical coordinates of client computer200on the surface of the Earth, which typically outputs a location as latitude and longitude values. GPS transceiver258can also employ other geo-positioning mechanisms, including, but not limited to, triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Enhanced Timing Advance (ETA), Base Station Subsystem (BSS), or the like, to further determine the physical location of client computer200on the surface of the Earth. It is understood that under different conditions, GPS transceiver258can determine a physical location for client computer200. In one or more embodiments, however, client computer200may, through other components, provide other information that may be employed to determine a physical location of the client computer, including for example, a Media Access Control (MAC) address, IP address, and the like.

In at least one of the various embodiments, applications, such as, operating system206, other client apps224, web browser226, or the like, may be arranged to employ geo-location information to select one or more localization features, such as, time zones, languages, currencies, calendar formatting, or the like. Localization features may be used in user-interfaces, reports, as well as internal processes or databases. In at least one of the various embodiments, geo-location information used for selecting localization information may be provided by GPS258. Also, in some embodiments, geolocation information may include information provided using one or more geolocation protocols over the networks, such as, wireless network108or network111.

Human interface components can be peripheral devices that are physically separate from client computer200, allowing for remote input or output to client computer200. For example, information routed as described here through human interface components such as display250or keyboard252can instead be routed through network interface232to appropriate human interface components located remotely. Examples of human interface peripheral components that may be remote include, but are not limited to, audio devices, pointing devices, keypads, displays, cameras, projectors, and the like. These peripheral components may communicate over networks implemented using WiFi, Bluetooth™, Bluetooth LTE™, and the like. One non-limiting example of a client computer with such peripheral human interface components is a wearable computer, which might include a remote pico projector along with one or more cameras that remotely communicate with a separately located client computer to sense a user's gestures toward portions of an image projected by the pico projector onto a reflected surface such as a wall or the user's hand.

Memory204may include RAM, ROM, or other types of memory. Memory204illustrates an example of computer-readable storage media (devices) for storage of information such as computer-readable instructions, data structures, program modules or other data. Memory204may store BIOS208for controlling low-level operation of client computer200. The memory may also store operating system206for controlling the operation of client computer200. It will be appreciated that this component may include a general-purpose operating system such as a version of UNIX, or Linux®, or a specialized client computer communication operating system such as Windows Phone™, or the Symbian® operating system. The operating system may include, or interface with a Java virtual machine module that enables control of hardware components or operating system operations via Java application programs.

Memory204may further include one or more data storage210, which can be utilized by client computer200to store, among other things, applications220or other data. For example, data storage210may also be employed to store information that describes various capabilities of client computer200. The information may then be provided to another device or computer based on any of a variety of methods, including being sent as part of a header during a communication, sent upon request, or the like. Data storage210may also be employed to store social networking information including address books, buddy lists, aliases, user profile information, or the like. Data storage210may further include program code, data, algorithms, and the like, for use by a processor, such as processor202to execute and perform actions. In one embodiment, at least some of data storage210might also be stored on another component of client computer200, including, but not limited to, non-transitory processor-readable removable storage device236, processor-readable stationary storage device234, or even external to the client computer.

Applications220may include computer executable instructions which, when executed by client computer200, transmit, receive, or otherwise process instructions and data. Applications220may include, for example, client user interface engine222, other client applications224, web browser226, or the like. Client computers may be arranged to exchange communications one or more servers.

Other examples of application programs include calendars, search programs, email client applications, IM applications, SMS applications, Voice Over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, and so forth.

Additionally, in one or more embodiments (not shown in the figures), client computer200may include an embedded logic hardware device instead of a CPU, such as, an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), Programmable Array Logic (PAL), or the like, or combination thereof. The embedded logic hardware device may directly execute its embedded logic to perform actions. Also, in one or more embodiments (not shown in the figures), client computer200may include one or more hardware micro-controllers instead of CPUs. In one or more embodiments, the one or more micro-controllers may directly execute their own embedded logic to perform actions and access its own internal memory and its own external Input and Output Interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as System On a Chip (SOC), or the like.

Illustrative Network Computer

FIG.3shows one embodiment of network computer300that may be included in a system implementing one or more of the various embodiments. Network computer300may include many more or less components than those shown inFIG.3. However, the components shown are sufficient to disclose an illustrative embodiment for practicing these innovations. Network computer300may represent, for example, one or more embodiments of a predictive learner server computer such as predictive learner server computer116, or the like, ofFIG.1.

Network computers, such as, network computer300may include a processor302that may be in communication with a memory304via a bus328. In some embodiments, processor302may be comprised of one or more hardware processors, or one or more processor cores. In some cases, one or more of the one or more processors may be specialized processors designed to perform one or more specialized actions, such as, those described herein. Network computer300also includes a power supply330, network interface332, audio interface356, display350, keyboard352, input/output interface338, processor-readable stationary storage device334, and processor-readable removable storage device336. Power supply330provides power to network computer300.

Network interface332includes circuitry for coupling network computer300to one or more networks, and is constructed for use with one or more communication protocols and technologies including, but not limited to, protocols and technologies that implement any portion of the Open Systems Interconnection model (OSI model), global system for mobile communication (GSM), code division multiple access (CDMA), time division multiple access (TDMA), user datagram protocol (UDP), transmission control protocol/Internet protocol (TCP/IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), general packet radio service (GPRS), WAP, ultra-wide band (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMax), Session Initiation Protocol/Real-time Transport Protocol (SIP/RTP), or any of a variety of other wired and wireless communication protocols. Network interface332is sometimes known as a transceiver, transceiving device, or network interface card (NIC). Network computer300may optionally communicate with a base station (not shown), or directly with another computer.

Audio interface356is arranged to produce and receive audio signals such as the sound of a human voice. For example, audio interface356may be coupled to a speaker and microphone (not shown) to enable telecommunication with others or generate an audio acknowledgment for some action. A microphone in audio interface356can also be used for input to or control of network computer300, for example, using voice recognition.

Display350may be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LED), Organic LED (OLED) or any other type of light reflective or light transmissive display that can be used with a computer. In some embodiments, display350may be a handheld projector or pico projector capable of projecting an image on a wall or other object.

Network computer300may also comprise input/output interface338for communicating with external devices or computers not shown inFIG.3. Input/output interface338can utilize one or more wired or wireless communication technologies, such as USB™, Firewire™, WiFi, WiMax, Thunderbolt™, Infrared, Bluetooth™, Zigbee™, serial port, parallel port, and the like.

Also, input/output interface338may also include one or more sensors for determining geolocation information (e.g., GPS), monitoring electrical power conditions (e.g., voltage sensors, current sensors, frequency sensors, and so on), monitoring weather (e.g., thermostats, barometers, anemometers, humidity detectors, precipitation scales, or the like), or the like. Sensors may be one or more hardware sensors that collect or measure data that is external to network computer300. Human interface components can be physically separate from network computer300, allowing for remote input or output to network computer300. For example, information routed as described here through human interface components such as display350or keyboard352can instead be routed through the network interface332to appropriate human interface components located elsewhere on the network. Human interface components include any component that allows the computer to take input from, or send output to, a human user of a computer. Accordingly, pointing devices such as mice, styluses, track balls, or the like, may communicate through pointing device interface358to receive user input.

GPS transceiver340can determine the physical coordinates of network computer300on the surface of the Earth, which typically outputs a location as latitude and longitude values. GPS transceiver340can also employ other geo-positioning mechanisms, including, but not limited to, triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Enhanced Timing Advance (ETA), Base Station Subsystem (BSS), or the like, to further determine the physical location of network computer300on the surface of the Earth. It is understood that under different conditions, GPS transceiver340can determine a physical location for network computer300. In one or more embodiments, however, network computer300may, through other components, provide other information that may be employed to determine a physical location of the client computer, including for example, a Media Access Control (MAC) address, IP address, and the like.

In at least one of the various embodiments, applications, such as, operating system306, skills engine322, ingestion engine324, other services328, or the like, may be arranged to employ geo-location information to select one or more localization features, such as, time zones, languages, currencies, currency formatting, calendar formatting, or the like. Localization features may be used in user interfaces, dashboards, reports, as well as internal processes or databases. In at least one of the various embodiments, geo-location information used for selecting localization information may be provided by GPS340. Also, in some embodiments, geolocation information may include information provided using one or more geolocation protocols over the networks, such as, wireless network108or network111.

Memory304may include Random Access Memory (RAM), Read-Only Memory (ROM), or other types of memory. Memory304illustrates an example of computer-readable storage media (devices) for storage of information such as computer-readable instructions, data structures, program modules or other data. Memory304stores a basic input/output system (BIOS)308for controlling low-level operation of network computer300. The memory also stores an operating system306for controlling the operation of network computer300. It will be appreciated that this component may include a general-purpose operating system such as a version of UNIX, or Linux®, or a specialized operating system such as Microsoft Corporation's Windows® operating system, or the Apple Corporation's MacOS® operating system. The operating system may include, or interface with one or more virtual machine modules, such as, a Java virtual machine module that enables control of hardware components or operating system operations via Java application programs. Likewise, other runtime environments may be included.

Memory304may further include one or more data storage310, which can be utilized by network computer300to store, among other things, applications320or other data. For example, data storage310may also be employed to store information that describes various capabilities of network computer300. The information may then be provided to another device or computer based on any of a variety of methods, including being sent as part of a header during a communication, sent upon request, or the like. Data storage310may also be employed to store social networking information including address books, friend lists, aliases, user profile information, or the like. Data storage310may further include program code, data, algorithms, and the like, for use by a processor, such as processor302to execute and perform actions such as those actions described below. In one embodiment, at least some of data storage310might also be stored on another component of network computer300, including, but not limited to, non-transitory media inside processor-readable removable storage device336, processor-readable stationary storage device334, or any other computer-readable storage device within network computer300, or even external to network computer300. Data storage310may include, for example, course information312, course profiles314, skills models316, cognitive taxonomies318, or the like.

Applications320may include computer executable instructions which, when executed by network computer300, transmit, receive, or otherwise process messages (e.g., SMS, Multimedia Messaging Service (MMS), Instant Message (IM), email, or other messages), audio, video, and enable telecommunication with another user of another mobile computer. Other examples of application programs include calendars, search programs, email client applications, IM applications, SMS applications, Voice Over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, and so forth. Applications320may include skills engine322, ingestion engine324, other services328, or the like, that may be arranged to perform actions for embodiments described below. In one or more of the various embodiments, one or more of the applications may be implemented as modules or components of another application. Further, in one or more of the various embodiments, applications may be implemented as operating system extensions, modules, plugins, or the like.

Furthermore, in one or more of the various embodiments, skills engine322, ingestion engine324, other services328, or the like, may be operative in a cloud-based computing environment. In one or more of the various embodiments, these applications, and others, that comprise the management platform may be executing within virtual machines or virtual servers that may be managed in a cloud-based based computing environment. In one or more of the various embodiments, in this context the applications may flow from one physical network computer within the cloud-based environment to another depending on performance and scaling considerations automatically managed by the cloud computing environment. Likewise, in one or more of the various embodiments, virtual machines or virtual servers dedicated to skills engine322, ingestion engine324, other services328, or the like, may be provisioned and de-commissioned automatically.

Also, in one or more of the various embodiments, skills engine322, ingestion engine324, other services328, or the like, may be located in virtual servers running in a cloud-based computing environment rather than being tied to one or more specific physical network computers.

Further, network computer300may also comprise hardware security module (HSM)360for providing additional tamper resistant safeguards for generating, storing or using security/cryptographic information such as, keys, digital certificates, passwords, passphrases, two-factor authentication information, or the like. In some embodiments, hardware security module may be employed to support one or more standard public key infrastructures (PKI), and may be employed to generate, manage, or store keys pairs, or the like. In some embodiments, HSM360may be a stand-alone network computer, in other cases, HSM360may be arranged as a hardware card that may be installed in a network computer.

Additionally, in one or more embodiments (not shown in the figures), network computer300may include an embedded logic hardware device instead of a CPU, such as, an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), Programmable Array Logic (PAL), or the like, or combination thereof. The embedded logic hardware device may directly execute its embedded logic to perform actions. Also, in one or more embodiments (not shown in the figures), the network computer may include one or more hardware microcontrollers instead of a CPU. In one or more embodiments, the one or more microcontrollers may directly execute their own embedded logic to perform actions and access their own internal memory and their own external Input and Output Interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as System On a Chip (SOC), or the like.

Illustrative Logical System Architecture

FIG.4illustrates a logical architecture of system400for determining interpersonal or behavioral skills based on course information in course offerings in accordance with one or more of the various embodiments.

In some cases, it may be advantageous for educational institutions to determine the skills and skill proficiency that may be learned or taught by courses offered by the institutions. Also, it may be advantageous for educational organizations, government agencies, or the like, to compare different courses, skills, skill proficiency, or the like, across educational organizations. Likewise, in some cases, it may be advantageous for students or others to evaluate courses, courses of study, institutions, or the like, based on the skills and skill proficiencies that may be learned from particular courses.

Accordingly, in some embodiments, skills engines may be arranged to provide a normalized/standard representation of the skills and skill proficiencies of courses taught by institutions. Conventionally, institutions may provide lists or summaries or skill and proficiencies for various courses. However, often such lists or summaries may comprise non-standard skill definitions, arbitrary skill proficiencies, or the like. Likewise, there may be mismatches between skill summaries and the actual teaching/learning that may occur in specific courses.

In some cases, students or others may employ course information such as syllabuses, course catalog descriptions, or the like, to evaluate skill or skill proficiencies that may be learned from various courses. However, in some cases, determining skill and skill proficiencies from course information may be disadvantageous for one or more reasons, including lack of standardization within or across institutions, mismatches in the course information and the actual course, or the like.

While, in some cases, learners or employers may be enabled to determine particular skills or acumen provided by various courses based on a review of conventional course information. Often this course information may provide detailed descriptions of various learning objectives for a course. However, often the learning objectives narrowly describe the subjects being taught as well as specific activities student may perform to demonstrate their learned skills. Also, conventional course information typically is focused on describing tangible skills or tangible activities that may be learned or performed.

Also, in many cases, conventional course information may omit describing important soft-skills that may be learned or acquired from a course. Accordingly, learners or employers may be unable to discern which soft-skills may be associated with a given course. However, in some cases, the natural language included in narratives associated with learning objectives may be evaluated to identify if a course may convey soft-skills.

Accordingly, in some embodiments, systems, such as, system400may include skills engines that may be arranged to infer the soft skills that may be learned from course.

In one or more of the various embodiments, system400may be arranged to include skills engine402, soft skills models404, course information406, course profile408, or the like.

In one or more of the various embodiments, skills engines, such as, skills engine402may be arranged to determine one or more soft-skills that may be acquired from courses based on the course information associated with a course. In some embodiments, course information may include information, such as, course syllabuses, course catalogs, course descriptions, or the like. In some embodiments, course information may be collected from one or more sources and stored in a course information data store (not shown). In some embodiments, course information may be considered information generally available to the public (e.g., learners or employers) that describe one or more learning objectives of a course. In many cases, learning objectives may be described with a text-based narrative that describe the learning activities, subject matter, or the like. Often a course may include more than one learning objective.

In one or more of the various embodiments, skills engines may be arranged to employ one or more skills models, such as, skills models404, that employ natural language processing, machine learning, various heuristics, or the like, to determine soft skills based on the learning objectives for a course.

In some embodiments, skills engines may be arranged to determine the soft-skills that may be inferred to be learned from those learning objectives. Accordingly, in some embodiments, skills engines may be arranged to update course profiles to reflect the soft-skills that may be learned or acquired from courses taken. Thus, in some embodiments, learners and employers may be provided a better understanding of how a particular course may satisfy their goals.

FIG.5illustrates a logical representation of system500for determining interpersonal or behavioral skills based on course information in accordance with one or more of the various embodiments. As described above, in some embodiments, skills engines, such as, skills engine502may be provided learning objectives, such as, learning objective506, from course information510. Accordingly, in some embodiments, skills engine502may employ a skills model, such as, skills model504to infer which soft-skills, such as, soft-skills508may be associated with learning objective506.

In one or more of the various embodiments, learning objectives, such as, learning objective506may be determined from course information, such as, course information510. In some embodiments, one or more services, such as, ingestion engine512may be arranged to employ machine-learning, NLP, heuristics, or the like, to identify the learning objectives from course information. For example, course syllabuses often include one or more sections dedicated to outlining the learning objectives for a course. In some cases, educational institutions may employ a consistent format or structured format for their course information that enable ingestion engines to extract learning objectives.

Also, in some embodiments, learning objectives may be a chunk of text that provides description of activities, projects, subject matter, or the like in a natural language narrative that ingestion engines may be trained or configured to recognize or extract. Accordingly, in some embodiments, ingestion engines may be arranged to employ extraction models, such as, extraction model514for determining learning objectives or learning objective narratives from course information. In some embodiments, ingestion engines or predictive learner platforms may be arranged to determine extraction models based on configuration information. Thus, in some embodiments, different extraction models may be provided for different organizations to account for local requirements or local circumstances.

In some embodiments, skills engines may be arranged to receive learning objectives in text form and employ one or more skills models to infer the soft-skills that may be conferred by the learning objective, if any.

FIG.6illustrates a logical schematic of soft-skills700for determining interpersonal or behavioral skills based on course information in accordance with one or more of the various embodiments.

In some embodiments, skills engines may be arranged to infer one or more soft-skills from learning objectives. Soft-skills may be a class of generally applicable skills that may be learned from various courses of differing subjects. In some cases, acquiring soft-skills may be the indirect result of the actions or activities associated with learning one or more hard-skills. In this example, table602represents a collection of soft-skills. In some cases, one or more soft skills may be considered major skills that may be associated with one or more other soft-skills, such as, collection606or collection608.

FIG.7illustrates a logical schematic of system700for determining interpersonal or behavioral skills based on course information in accordance with one or more of the various embodiments. As described above, skills engines may be arranged to support more than one skills model. Thus, in some embodiments, skills engines may be arranged to employ new or improved or skills models if they may be discovered.

In some embodiments, skills engines may be arranged to determine learning objective narratives that comprise text. Accordingly, in some embodiments, skills models may be arranged to include one or more of heuristics, machine-learning classifiers, NLP, or the like, for determining soft-skills, cognitive levels, or the like.

In this example, for some embodiments, skills model704may be arranged to determine one or more soft-skills that may be associated with a learning objective based on one or more provided learning objective narratives. Accordingly, in some embodiments, skills engines may be arranged to provide one or more learning objective narratives, such as, learning objective narratives702to a skills model, such as skills model704.

In some embodiments, skills models, such as, skills model704may be arranged to employ NLP to determine verbs and objects from the learning objective narrative. Accordingly, in some embodiments, the verb and object tuples may be mapped to one or more soft-skills as illustrated by table706. In some embodiments, skills models may employ one set of heuristics, machine-learning classifiers, or NLP to determine the soft-skills. Further, in some embodiments, a dedicated training engine (not shown) or skills engine, such as, skills engine322may be arranged to train one or more machine learning based skills models. In one or more of the various embodiments, the particular actions performed for training skills models may depend on type of model (e.g., linear regression, deep learning, deep neural networks, decision trees, ensembles of two or more smaller models, or the like). Also in some embodiments, the type of training may be dependent on the application the skills models may be targeted towards. For example, in one or more of the various embodiments, for some classification problems one ML model type may be preferred over another. Accordingly, in some embodiments, skills engines may be arranged to collect feedback from users or administrators that may be used to score or rate the performance of various skills models. Accordingly, in some embodiments, these innovations anticipate that skills models may be changed or improved often or continuously based on user/administrator feedback, advancements in ML/prediction models, changes to available training data, or the like.

Also, in one or more of the various embodiments, skills models may be arranged to infer cognitive skills or cognitive levels as indicated by column706.

In one or more of the various embodiments, skills engines may be arranged to provide user interfaces that enable learners, employers, or other users to evaluate how well a skills model performed. Accordingly, in some embodiments, organizations may collect feedback regarding the performance of skills models. Accordingly, in some embodiments, if the feedback for a skills model results in a quality score that falls below a threshold value, skills engines may be arranged to automatically withdraw that skills model from production.

FIG.8illustrates a logical schematic of syllabus800for determining interpersonal or behavioral skills based on course information in accordance with one or more of the various embodiments. In some cases, course information may be provided by syllabuses, such as syllabus800. Often syllabuses come in a variety of formats or arrangements. However, in most cases, syllabuses will include a collection of learning objectives for a given course. In some cases, the learning objectives are explicitly provided in a list, grouped into one or more sections, or the like. Alternatively, in some cases, learning objectives may be embedded in multiple sections of a syllabus. In this example, syllabus800includes section802, section804, section806, section808, section810, or the like. Accordingly, in this example, for some embodiments, one or more learning objectives may be included in one or more of these sections.

In some embodiments, ingestion engines may be arranged to extract learning objectives or learning objective narratives from course information, including syllabus800, or the like. Accordingly, in some embodiments, skills engines may be provided learning objective narratives from differently formatted or arranged syllabuses.

Generalized Operations

FIGS.9-12represent generalized operations for determining interpersonal or behavioral skills based on course information in accordance with one or more of the various embodiments. In one or more of the various embodiments, processes900,1000,1100, and1200described in conjunction withFIGS.9-12may be implemented by or executed by one or more processors on a single network computer, such as network computer300ofFIG.3. In other embodiments, these processes, or portions thereof, may be implemented by or executed on a plurality of network computers, such as network computer300ofFIG.3. In yet other embodiments, these processes, or portions thereof, may be implemented by or executed on one or more virtualized computers, such as those in a cloud-based environment. However, embodiments are not so limited and various combinations of network computers, client computers, or the like may be utilized. Further, in one or more of the various embodiments, the processes described in conjunction withFIGS.9-12may perform actions for interpersonal or behavioral skills based on course information in accordance with at least one of the various embodiments or architectures such as those described in conjunction withFIGS.4-8. Further, in one or more of the various embodiments, some or all of the actions performed by processes900,1000,1100, and1200may be executed in part by skills engine322, ingestion engine324, or the like.

FIG.9illustrates an overview flowchart for process900for determining interpersonal or behavioral skills based on course information in accordance with one or more of the various embodiments. After a start block, at block902, in one or more of the various embodiments, course information for a course may be provided. As described above, predictive learner platform may include or employ ingestion engines that may consume course information, such as, syllabuses, course catalog descriptions, or the like.

At block904, in one or more of the various embodiments, ingestion engines or skills engines may be arranged to determine one or more learning objectives from the course information. As described above, in some embodiments, ingestion engines or skills engines may be arranged to employ one or more extraction models that may include one or more heuristics, machine-learning classifiers, NLP, or the like, that may be employed to extract learning objectives and learning objective narratives from the course information. In some embodiments, one or more extraction models may be developed for different institutions or different types of course information.

In one or more of the various embodiments, learning objectives may include narratives that describe in human readable text the subjects the learning objective may be directed toward.

At block906, in one or more of the various embodiments, skills engines may be arranged to determine one or more soft skills based on the learning objectives or other course information that may be included in course syllabuses, or the like. In some embodiments, skills engines may be arranged to provide learning objective narratives to one or more skills models that may be trained or tuned to infer soft-skills from course information.

At block908, in one or more of the various embodiments, skills engines may be arranged to be associate one or more determined soft-skills with the course profile for the course. Accordingly, in some embodiments, predictive learner platforms may be arranged to employ these enhanced course profiles to match learners with employment opportunities. Likewise, in some embodiments, updated course profiles may be employed to determine learning pathways for students seeking particular career or employment opportunity.

FIG.10illustrates a flowchart for process1000for determining interpersonal or behavioral skills based on course information in accordance with one or more of the various embodiments. After a start block, at block1002, in one or more of the various embodiments, a learning objective narrative may be provided to a skills engine. As described above, in some embodiments, learning objective narratives may be extracted from course information.

At block1004, in one or more of the various embodiments, optionally, skills engines may be arranged to determine a soft skills model. In some embodiments, skills engines may be configured to have multiple skills models. In some embodiments, one or more skills models may be trained for inferring soft-skills from learning objectives. In some cases, different skills models may be tuned for different types of skills. Also, in some embodiments, one or more skills models may be tuned for different organizations, education institutions, career areas, or the like. For example, in some embodiments, educational institution A may employ formats or standards for course information that may be significantly different than educational institution B where it may be advantageous to have separately tuned/trained skills models. Accordingly, in some embodiments, skills engines may be arranged to employ rules, instructions, or the like, that associate particular skills models with particular education institutions.

Note, this block is indicated as being optional because in some cases for some embodiments skills engines may be configured to use a default skills model or otherwise have determined the skills model previously.

At block1006, in one or more of the various embodiments, skills engines may be arranged to determine one or more soft-skills based on the learning objective.

In one or more of the various embodiments, one or more skills models may be arranged to compare the similarity of learning objective narratives with narratives, word sets, or the like, that may be associated with one or more soft skills. For example, in some embodiments, portions of learning objective narratives may be vectorized to enable skills models that support cosine similarity evaluation, or the like. Likewise, in some embodiments, skills models may be configured to perform NLP such as word/phrase frequency analysis, parts of speech comparisons, or the like, to determine similarity scores for learning objectives. In some embodiments, skills models may combine two or more NLP methods for correlating one or more soft skills with learning objectives.

At block1008, in one or more of the various embodiments, skills engines may be arranged to associate the one or more determined soft-skills with the course profile of the course.

FIG.11illustrates a flowchart for process1100for determining interpersonal or behavioral skills based on course information in accordance with one or more of the various embodiments. After a start block, at block1102, in one or more of the various embodiments, a learning objective narrative may be provided to a skills engine. As described above, in some embodiments, skills engines may provide learning objectives narratives that may be determined from course information.

At block1104, in one or more of the various embodiments, optionally skills engines may be arranged to determine a skills model for evaluating the learning objective narrative.

In some embodiments, skills engines may be configured with multiple skills models available. In some embodiments, one or more skills models may be trained for inferring soft-skills. In some cases, different skills models may be tuned for different classes of soft skills. Also, in some embodiments, one or more skills models may be tuned for different organizations, education institutions, career areas, or the like. For example, in some embodiments, educational institution A may employ formats or standards for course information that may be significantly different than educational institution B where it may be advantageous to have separately tuned skills models. Accordingly, in some embodiments, skills engines may be arranged to employ rules, instructions, or the like, that associate particular skills models with particular education institutions.

Here, in some embodiments, skills engines may be arranged to determine a skills model that may be trained or tuned for determining soft-skills directly from learning objective narratives.

Note, this block is indicated as being optional because in some cases for some embodiments skills engines may be configured to use a default skills model or otherwise have determined the skills model previously.

At block1106, in one or more of the various embodiments, skills engines may be arranged to determine one or more soft-skills associated with the learning objective based on a skills model. In one or more of the various embodiments, skills engines may be arranged to employ the determined skills models to infer one or more soft-skills from the learning objective narrative. Accordingly, in some embodiments, skills engines may execute one or more NLP actions as per the skills model to determine one or more soft-skills that may be associated with the learning objective.

As described herein, in one or more of the various embodiments, skills models may be arranged to base soft-skills inferences on NLP matches of portions of the learning objective narrative with a soft-skills taxonomy that associates various words or phrases with particular soft-skills.

At block1108, in one or more of the various embodiments, skills engines may be arranged to associate the one or more soft-skills with the course profile. In one or more of the various embodiments, the one or more inferred soft-skills may be included in course profile of the course associated with the learning objective being evaluated.

FIG.12illustrates a flowchart for process1200for determining interpersonal or behavioral skills based on course information in accordance with one or more of the various embodiments. After a start block, at block1202, in one or more of the various embodiments, skills engines may be arranged to employ skills models to infer one or more soft-skills, or the like, from course information. In some embodiments, course profiles may be updated based on the inferred soft-skills, or the like. Accordingly, in some embodiments, course profile information may be employed to update learner profiles based on the soft-skills, or the like, that may be learned from various courses.

At block1204, in one or more of the various embodiments, skills engines may be arranged to collect feedback information. In one or more of the various embodiments, skills engines, predictive learner platforms, or the like, may be arranged to provide various user interfaces that enable users, learners, employers, administrators, or the like, to provide direct or indirect regarding the veracity of the inferences. Accordingly, in some embodiments, skills engines may be arranged to collect one or more quality metrics associated with particular skills models. For example, in some embodiments, organizations may employ one or more users to evaluate a sample of inferences made by skills models. Also, for example, in some embodiments, learners or employers may be provided user interfaces to collect direct feedback related to learner-role matches, or the like. Also, in some embodiments, over time predictive learner platforms may be enabled to track the employment history of matched learners. Thus, for example, predictive learner platforms may be arranged to track the duration of employment of matched learners. Likewise, in some embodiments, predictive learner platforms may be arranged to monitor how matched learners fail to convert into employees of matched employers.

At block1206, in one or more of the various embodiments, skills engines may be arranged to determine quality score for skills model based on collected feedback information. In some embodiments, predictive learner platforms may be arranged to employ one or more rules, instructions, formulas, or the like, provided via configuration information to evaluate the one or more metrics collected from feedback information. Accordingly, in some embodiments, one or more metric values may be combined into one or more quality scores that may be employed to determine if skills models may be provided quality inferences.

In one or more of the various embodiments, skills engines may be arranged to compare the performance of skills models with one or more reference models. Accordingly, in one or more of the various embodiments, skills engines may be arranged to provide test information from a corpus of course information for one or more exemplar courses. Thus, in some embodiments, skills engines may be arranged to automatically generate quality scores for the skills models.

Further, in some embodiments, skills engines may be arranged to generate quality scores for skills models based on a combination of quality metrics or machine learning evaluation using reference models. In some embodiments, different types of quality metrics, reference models, or the like, may be employed for different types of courses, different cognitive levels, different soft skills, or the like. Thus, in some embodiments, skills engines may be arranged to determine particular quality metrics, reference models, quality determinations, or the like, based on configuration information to account for local circumstances or local requirements.

At decision block1208, in one or more of the various embodiments, if one or more quality scores may be below a threshold value, control may flow to block1210; otherwise, control may be returned to a calling process.

In one or more of the various embodiments, quality scores of skills models may fall overtime if the narratives associated with learning objectives change overtime. Also, in some embodiments, as predictive learner platforms may be applied to new organizations, new subject matter, or the like, inference veracity of skills models may decline. Similarly, in some embodiments, the meaning or application of one or more terms used in learning objectives may shift overtime which may cause skills model veracity to decline.

Accordingly, in some embodiments, if a skills model may be associated with low quality scores, skills engines or predictive learner platforms may be arranged to remove such skills models from production.

At block1210, in one or more of the various embodiments, the low quality skills model may be retrained. In some embodiments, predictive learner platforms may be arranged to automatically provide skills models to automatic or machine assisted training systems to retrain the low quality skills models. For example, in some embodiments, skills engines may retrain low-quality skills models using one or more machine learning reference models and a corpus of training data course information.

Alternatively, in some embodiments, predictive learner platforms may be arranged to discard low quality models in lieu of new skills models that may perform at acceptable levels.

Further, in one or more embodiments (not shown in the figures), the logic in the illustrative flowcharts may be executed using an embedded logic hardware device instead of a CPU, such as, an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), Programmable Array Logic (PAL), or the like, or combination thereof. The embedded logic hardware device may directly execute its embedded logic to perform actions. In one or more embodiments, a microcontroller may be arranged to directly execute its own embedded logic to perform actions and access its own internal memory and its own external Input and Output Interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as System On a Chip (SOC), or the like.