System and Method for Deploying Customized Large Language Models to Assist Individuals with Disabilities

A method, computer program product, and computing system for processing a query from a user on a website. The query is classified to determine whether the query is associated with a user with a disability. In response to classifying the query as being associated with a user with a disability, the query is processed using a customized large language model (LLM).

BACKGROUND

Approximately one billion people (15% of the world's population) live with a disability. According to the U.S. Center for Disease Control, 10.9% of U.S. adults experience cognition impairment, 5.7% experience hearing impairment, and 4.9% experience vision impairment. Article 21 of the United Nations Convention on the Rights of Persons with Disabilities defines access to information as a human right. Providing relevant information to individuals with disabilities in accessible formats and technologies that are appropriate to different kinds of disabilities, in a timely manner and without additional cost, is an important step in supporting this right. From a business perspective, increased accessibility enhances a company brand, extends market reach, and reduces legal risk.

Accessing information resources is difficult for individuals with disabilities. For example, when searching for a particular product or service, or for a feature or function that can help the individual, usability challenges arise. As an example, consider a person looking for a “laptop for hearing impaired.” In this example, a user may use a search engine to generate a query (e.g., “laptop for hearing impaired”). A resulting search engine result may include a page full of links (the top ones are paid advertisements), none of which provides a clear and useful answer to the question, and all of which require additional browsing and comprehension.

SUMMARY OF DISCLOSURE

In one example implementation, a computer-implemented method executed on a computing device may include, but is not limited to, processing a query from a user on a website. The query is classified to determine whether the query is associated with a user with a disability. In response to classifying the query as being associated with a user with a disability, the query is processed using a customized large language model (LLM).

One or more of the following example features may be included. In response to classifying the query as not being associated with a user with a disability, the query is processed using a query processing engine associated with the website. Classifying the query as being associated with a user with a disability may include comparing one or more portions of the query against a database of phrases associated with disabilities. Classifying the query as being associated with a user with a disability may include processing the query using a classification-based machine learning model. A particular customized LLM may be identified based upon, at least in part, a classification associated with the query. Classifying the query as being associated with a user with a disability may include one or more of: performing a binary classification associated with the query, and performing a multi-class classification associated with the query. A result may be generated for the query using the customized LLM. Feedback from the user concerning the generated result may be processed to update the classifying of subsequent queries.

In another example implementation, a computer program product resides on a computer readable medium that has a plurality of instructions stored on it. When executed by a processor, the instructions cause the processor to perform operations that may include, but are not limited to, processing a query from a user on a website. The query is classified to determine whether the query is associated with a user with a disability. In response to classifying the query as being associated with a user with a disability, the query is processed using a customized large language model (LLM).

One or more of the following example features may be included. In response to classifying the query as not being associated with a user with a disability, the query is processed using a query processing engine associated with the website. Classifying the query as being associated with a user with a disability may include comparing one or more portions of the query against a database of phrases associated with disabilities. Classifying the query as being associated with a user with a disability may include processing the query using a classification-based machine learning model. A particular customized LLM may be identified based upon, at least in part, a classification associated with the query. Classifying the query as being associated with a user with a disability may include one or more of: performing a binary classification associated with the query, and performing a multi-class classification associated with the query. A result may be generated for the query using the customized LLM. Feedback from the user concerning the generated result may be processed to update the classifying of subsequent queries.

In another example implementation, a computing system includes at least one processor and at least one memory architecture coupled with the at least one processor, wherein the at least one processor configured to process a query from a user on a website. The query is classified to determine whether the query is associated with a user with a disability. In response to classifying the query as being associated with a user with a disability, the query is processed using a customized large language model (LLM).

One or more of the following example features may be included. In response to classifying the query as not being associated with a user with a disability, the query is processed using a query processing engine associated with the website. Classifying the query as being associated with a user with a disability may include comparing one or more portions of the query against a database of phrases associated with disabilities. Classifying the query as being associated with a user with a disability may include processing the query using a classification-based machine learning model. A particular customized LLM may be identified based upon, at least in part, a classification associated with the query. Classifying the query as being associated with a user with a disability may include one or more of: performing a binary classification associated with the query, and performing a multi-class classification associated with the query. A result may be generated for the query using the customized LLM. Feedback from the user concerning the generated result may be processed to update the classifying of subsequent queries.

DETAILED DESCRIPTION

System Overview

Referring toFIG.1, there is shown disability access assistance process10that may reside on and may be executed by storage system12, which may be connected to network14(e.g., the Internet or a local area network). Examples of storage system12may include, but are not limited to: a Network Attached Storage (NAS) system, a Storage Area Network (SAN), a personal computer with a memory system, a server computer with a memory system, and a cloud-based device with a memory system.

As is known in the art, a SAN may include one or more of a personal computer, a server computer, a series of server computers, a mini computer, a mainframe computer, a RAID device and a NAS system. The various components of storage system12may execute one or more operating systems, examples of which may include but are not limited to: Microsoft® Windows®; Mac® OS X®; Red Hat® Linux®, Windows® Mobile, Chrome OS, Blackberry OS, Fire OS, or a custom operating system. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries or both; Mac and OS X are registered trademarks of Apple Inc. in the United States, other countries or both; Red Hat is a registered trademark of Red Hat Corporation in the United States, other countries or both; and Linux is a registered trademark of Linus Torvalds in the United States, other countries or both).

The instruction sets and subroutines of disability access assistance process10, which may be stored on storage device16included within storage system12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage system12. Storage device16may include but is not limited to: a hard disk drive; a tape drive; an optical drive; a RAID device; a random access memory (RAM); a read-only memory (ROM); and all forms of flash memory storage devices. Additionally/alternatively, some portions of the instruction sets and subroutines of disability access assistance process10may be stored on storage devices (and/or executed by processors and memory architectures) that are external to storage system12.

Various IO requests (e.g. IO request20) may be sent from client applications22,24,26,28to storage system12. Examples of IO request20may include but are not limited to data write requests (e.g., a request that content be written to storage system12) and data read requests (e.g., a request that content be read from storage system12).

The instruction sets and subroutines of client applications22,24,26,28, which may be stored on storage devices30,32,34,36(respectively) coupled to client electronic devices38,40,42,44(respectively), may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into client electronic devices38,40,42,44(respectively). Storage devices30,32,34,36may include but are not limited to: hard disk drives; tape drives; optical drives; RAID devices; random access memories (RAM); read-only memories (ROM), and all forms of flash memory storage devices. Examples of client electronic devices38,40,42,44may include, but are not limited to, personal computer38, laptop computer40, smartphone42, notebook computer44, a server (not shown), a data-enabled, cellular telephone (not shown), and a dedicated network device (not shown).

Users46,48,50,52may access storage system12directly through network14or through secondary network18. Further, storage system12may be connected to network14through secondary network18, as illustrated with link line54.

Client electronic devices38,40,42,44may each execute an operating system, examples of which may include but are not limited to Microsoft® Windows®; Mac® OS X®; Red Hat® Linux®, Windows® Mobile, Chrome OS, Blackberry OS, Fire OS, or a custom operating system. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries or both; Mac and OS X are registered trademarks of Apple Inc. in the United States, other countries or both; Red Hat is a registered trademark of Red Hat Corporation in the United States, other countries or both; and Linux is a registered trademark of Linus Torvalds in the United States, other countries or both).

In some implementations, as will be discussed below in greater detail, a data replication process, such as disability access assistance process10ofFIG.1, may include but is not limited to, processing a query from a user on a website. The query is classified to determine whether the query is associated with a user with a disability. In response to classifying the query as being associated with a user with a disability, the query is processed using a customized large language model (LLM).

For example purposes only, storage system12will be described as being a network-based storage system that includes a plurality of electro-mechanical backend storage devices. However, this is for example purposes only and is not intended to be a limitation of this disclosure, as other configurations are possible and are considered to be within the scope of this disclosure.

The Disability Access Assistance Process:

Referring also to the examples ofFIGS.2-3and in some implementations, disability access assistance process10may process300a query from a user on a website. The query is classified302to determine whether the query is associated with a user with a disability. In response to classifying the query as being associated with a user with a disability, the query is processed304using a customized large language model (LLM).

As will be discussed in greater detail below, implementations of the present disclosure may allow for advances in artificial intelligence, particularly generative artificial intelligence, to provide more effective responses to queries for individuals with disabilities. For example, people with disabilities often face a usability challenge when searching for a particular product or service, or for a feature or function that can help them when using a website or other Internet-connected browser. Using conventional browsers or other Internet-connected resources, these tools may be ineffective at providing meaningful responses to queries. For example, suppose a user (e.g., user46) provides a query for a “laptop for hearing impaired”. In this example, a conventional search engine may provide a page full of links, where the first links are paid advertisements as shown inFIG.3. Each of these links require additional browsing and contextual comprehension. Rather, these links fail to provide any clear or useful answers. As will be discussed in greater detail below, implementations of the present disclosure use a customized large language model (LLM) to provide contextual responses that are customized for responding to a disability identified from the query or other associated information (e.g., receiving devices, type of query, past query history, etc.). In this manner, disability access assistance process10can provide more meaningful answers to queries using a LLM that is customized to address queries concerning disabilities.

In some implementations, disability access assistance process10processing a query from a user on a website. A query may generally include a representation of a request from a user concerning a service or for information concerning a particular topic. For example, a query may include a question (e.g., “where can I purchase a laptop with hearing impairment features?”) or a text description for information (e.g., “best monitor settings to reduce overstimulation in ASD”). A query may also include multi-modal requests (e.g., a text description and a visual representation of a selected icon from a user interface; a speech signal with contextual information; a video file including a recording of a user asking a question; etc.). In each of these examples, the query may directly concern a disability (e.g., a text description for tablets and screens for those with visual impairments) or may be influenced by a disability (e.g., speech input may including vocal characteristics associated with a speech impediment). Accordingly, a query may be provided in various manners to a website or other resource to resolve the query. Referring also toFIG.4, a user (e.g., user46) may use a computing device (e.g., client electronic device38) to submit a query (e.g., query400) to a website. In one example, the website may include a specific website associated with a particular company or service. In another example, the website may be a general search engine or other browsing entry website from which users explore other websites.

In some implementations, disability access assistance process10classifies302the query to determine whether the query is associated with a user with a disability. Classifying302a query may generally include determining a class, topic, or type for the query. In some implementations, disability access assistance process10may classify302the query to determine whether the query is associated with or concerns a user with a disability. For example, classifying302the query may include processing the content of the query and/or contextual information associated with the source of the query. In one example, disability access assistance process10classifies302query400to determine whether query400is associated with a user living with a disability by processing the content of query400. In this example, suppose query400includes a text-based query in a search engine for “dell laptop for hearing impaired”. In this example and as will be discussed in greater detail below, disability access assistance process10processes query400for predefined words or phrases associated with different types of disabilities. Accordingly, disability access assistance process10may classify302query400based on the content of query400itself. In another example, disability access assistance process10may receive query400from a computing device (e.g., client electronic device38) with metadata or configuration information indicative of accessibility features associated with users with disabilities. For example, suppose disability access assistance process10determines from information associated with client electronic device38that client electronic device38has enabled hearing-impairment assistance functionality. In this example and subject to any privacy restrictions enabled by the user, disability access assistance process10may use contextual information (e.g., metadata of query400, metadata associated with client electronic device38, metadata with information concerning user46, etc.) to classify302query400as being associated with a user with a disability. As shown inFIG.4, the classification of query400is shown as classification module402. In some implementations, classification module402may include a hardware and/or software component configured to process query400and/or content associated with query400to classify302query400as being associated with a user with a disability.

In some implementations, classifying302the query as being associated with a user with a disability includes comparing306one or more portions of the query against a database of phrases associated with disabilities. Referring again toFIG.4, classification module402may include access to a database (e.g., database404). Database404may include a plurality of text, speech, and/or image portions that reference or concern disabilities. For example, database404may include text portions or keywords that may be indicative of a user with disabilities. Returning to the above example where user46provides query400with the text “dell laptop for hearing impaired”, disability access assistance process10classifies302query400as being associated with a user with a disability by comparing306one or more portions of query400(e.g., each word of query400) with predefined words from database400. Specifically, disability access assistance process10may classify302query400as being associated with a user with a hearing impairment based upon the comparison of the words “hearing impairment” with one or more entries of database404.

In some implementations, classifying302the query as being associated with a user with a disability includes processing308the query using a classification-based machine learning model. A machine learning model may generally include an algorithm or combination of algorithms that has been trained to recognize certain types of patterns. For example, machine learning approaches may be generally divided into three categories, depending on the nature of the signal available: supervised learning, unsupervised learning, and reinforcement learning. Supervised learning may include presenting a computing device with example inputs and their desired outputs, given by a “teacher”, where the goal is to learn a general rule that maps inputs to outputs. With unsupervised learning, no labels are given to the learning algorithm, leaving it on its own to find structure in its input. Unsupervised learning can be a goal in itself (discovering hidden patterns in data) or a means towards an end (feature learning). Reinforcement learning may generally include a computing device interacting in a dynamic environment in which it must perform a certain goal (such as driving a vehicle or playing a game against an opponent). As it navigates its problem space, the machine learning model is provided feedback that is analogous to rewards, which it tries to maximize. While three examples of machine learning approaches have been provided, it will be appreciated that other machine learning approaches are possible within the scope of the present disclosure.

In some implementations, a classification-based machine learning model (e.g., machine learning model426) is a machine learning model configured to classify data into classes based on certain characteristics of the data. For example, classifying302the query as being associated with a user with a disability may include one or more of: performing310a binary classification associated with the query, and performing312a multi-class classification associated with the query. In one example, machine learning model426may be trained to classify query400by performing310a binary classification. A binary classification is a classification of the query as being associated with a user with disability generally, or as not being associated with a user with disability. In this example and as will be discussed in greater detail below, disability access assistance process10may process304a query classified as being associated with a user with a disability by using a customized LLM that is more effective at being responsive to individuals with disabilities or more effective at providing relevant information concerning disabilities than conventional search engines used by many websites.

In some implementations, disability access assistance process10classifies302the query as being associated with a user with a disability includes performing312a multi-class classification associated with a query. A multi-class classification is a classification of the query into one of a plurality of classes. In this example, a machine learning model (e.g., machine learning model406) can classify302query400as being associated with a particular class of a plurality of classes. In one example, classes may be defined for different types of disabilities, different types of queries, and/or combinations thereof. In this manner and as will be discussed in greater detail below, disability access assistance process10may process304the query using a class-specific customized LLM.

In some implementations and in response to classifying the query as not being associated with a user with a disability, disability access assistance process10processes314the query using a query processing engine associated with the website. Referring again toFIG.4, suppose that disability access assistance process10classifies302query400as not being associated with a user with a disability. In this example, disability access assistance process10processes314query400using a query processing engine (e.g., query processing engine408) associated with a website or other resource. A query processing engine408may include a publicly or privately accessible search engine or algorithm that searches local or Internet-based resources based on the query. In this example, because the query is classified as not being associated with a user with a disability, query processing engine408may provide sufficiently relevant and understandable results (e.g., result410) to the user (e.g., user46).

In some implementations and in response to classifying the query as being associated with a user with a disability, disability access assistance process10processes304the query using a customized large language model (LLM). For example, instead of using a conventional search engine to address queries that are classified as being associated with a user with a disability, a recent alternative is the use of a large language model (LLM) such as ChatGPT. An LLM is a highly-compressed representation of the collective knowledge present on the Internet. LLMs are a particular type of neural network. The architecture (or structure) of such a network comprises many layers (called hidden layers) that connect the input layer to the output layer as shown inFIG.5. Values flow in this network from left to right. Every node in these layer has associated weights (denoted by the Wijand Biparameters) that define the calculation applied to the input values for this node in order to create the output values of the node, which in turn are passed on as input to nodes in the next layer. Essentially this calculation is a linear transformation of the input vector X to derive the output vector Y, i.e. Y=Wi(X)+Bi. The learning or training process of such a network works by comparing the current output of the network to the desired output (e.g., labeled examples) and then minimizing the gap (or error) through a process called back propagation, which adjusts the network weights in reverse order, from output to input. This optimization process is repeated until the error is sufficiently small.

In some implementations, large LLMs can have a huge number of parameters (i.e., the weights all of the nodes), with 175 billion in the case of GPT (i.e., the model underlying ChatGPT), and are often trained over many months on very large and highly curated datasets (e.g., 45 TB, or equivalently hundreds of billions of words, for ChatGPT,). The cost associated with building these models can be prohibitive, and thus the ability to use a pre-trained LLM offers considerable time and cost savings.

With fine-tuning, a pretrained neural network is customized or specialized for a specific task, which in the context of the present disclosure is to provide answers that are most useful for users with disabilities. This process typically requires a far smaller dataset (e.g., 100K words) and is therefore orders of magnitude faster and cheaper than full model training. Technically, fine-tuning works by changing the network weights so that the network is optimized (i.e., the error is minimized) for the specific task, or set of tasks, the LLM is being customized for.

In some implementations, customizing a LLM may involve a training set of a few thousand curated questions and answers. In principle, fine-tuning an LLM in such a way can affect not only the content of the answers, but also their format. While fine-tuning can provide very good results, it cannot be done with closed-source models such as ChatGPT, where the network weights are “frozen” and cannot be updated. Rather, in-context or few-shot learning through prompt design allows LLMs to be customized by taking a few task-specific examples (e.g., usually less than ten) as input, and then quickly figuring out how to perform well on that task. In some implementations, this process occurs as part of model inference, not training. No backpropagation takes place and the model weight are not changed. This allows closed-source models to be customized to particular applications (e.g., providing responses to queries associated with a user with a disability). In-context learning may seem like magic, and although it is understood to work by some combination of location-independent model parameters (i.e., known as attention) and location dependent parameters (i.e., known as positional embeddings), the inner working of in-context learning are not fully understood.

For example, disability access assistance process10may customize a general LLM via in-context learning for responding to queries for users with a disability by giving it a “prompt” (i.e., a query containing a short sequence of examples, each example having a question and matching answer). For example, this is shown inFIG.6. In some implementations, the in-context sequence may includes dozens to hundreds of relevant examples.

The above-described methods (i.e., fine-tuning and in-context learning) have tradeoffs between them. In-context learning is applicable even when the pre-trained model is “frozen” and can work with a very small number of examples. However, it increases the inference cost considerably, and since no learning takes place the customization has to be done for each individual query. Fine-tuning can be done once as an additional training phase and be used many times, and hence it is the preferable option for open-source models, if a large enough set of fine-tuning examples is available.

Referring again toFIG.4, an LLM (e.g., large language models412,414,416) often provides more concise and relevant answers than a search engine, albeit with a small risk that the answers may be incorrect. As an example, suppose disability access assistance process10classifies query400as being associated with a user with a disability. In this example, disability access assistance process10processes304query400with a customized LLM (e.g., LLM412).

In some implementations, disability access assistance process10may identify316a particular customized LLM based upon, at least in part, a classification associated with the query. For example, disability access assistance process10may identify316a customized LLM that is more specific to a particular query. Suppose LLM412is specialized for hearing impairment product recommendations, LLM414is customized for visual impairment product recommendations, and LLM416is customized for adapting electronic devices to users with developmental disabilities. Accordingly, when classifying302query400, disability access assistance process10may perform314multi-class classification on query400to determine that query400is associated with a user with a hearing impairment disability. Accordingly, disability access assistance process10processes304query400using LLM412.

In some implementations, disability access assistance process10generates318a result for the query using the customized LLM. For example, LLM412may provide interactive prompts to the user (e.g., user46) to generate result410. Result410may be updated with each additional prompt and response from LLM412. An example of result410generated by LLM412is shown inFIG.7. Result410may include a dialogue in text form, a text-to-speech audio output, and/or a visual result in the form of navigating a website. For example, in addition to describing a laptop that can help those with a hearing impairment, result410may include a script or other executable code to render relevant results in a browser on client electronic device38.

In some implementations, disability access assistance process10processes320feedback from the user concerning the generated result to update the classifying of subsequent queries. For example, as users interact with disability access assistance process10following such queries, disability access assistance process10tracks and measures the feedback provided by the user engagement. Typical metrics of engagement include time on site, pages per session, conversion rate or dollars spent, customer satisfaction (e.g. via survey), etc. By analyzing such metrics, disability access assistance process10determines the benefit of either the single LLM or multiple LLM approaches compared to the existing usage of a generic search engine to answer user queries.

In some implementations, disability access assistance process10can perform A/B testing by routing a certain percentage of the queries issued by disabled users (as determined by the initial classification model) to query processing engine408, and the others to the LLM(s) (e.g., LLMs412,414,416), and comparing the user engagement resulting from of these approaches.