Contextual image-assisted search

Methods, information handling systems and computer readable media are disclosed for a contextual image-based search. In one embodiment, a method includes receiving, at a computer implemented system for searching a knowledge base, an image submitted by a user of the system. The method continues with obtaining a classification label corresponding to the image, checking a context module database for the presence of a follow-up question associated with the classification label, and generating search term information for searching of the knowledge base, where the search term information comprises the classification label. If a follow-up question associated with the classification label is found, an embodiment further includes presenting the follow-up question to the user of the system, receiving from the user an answer to the follow-up question, modifying the search term information to include the answer, and checking the context module database for an additional follow-up question associated with the answer.

BACKGROUND

The present disclosure relates generally to networked information handling systems, and more particularly to systems for and methods of contextual image-assisted search of a knowledge base.

Use of an information handling system to access information may involve searching of an information store, or knowledge base. In some cases a user has limited familiarity with the subject matter of a knowledge base or the way the knowledge base is organized. This lack of familiarity can make it difficult to formulate an appropriate search query for obtaining the desired information.

SUMMARY

Methods, information handling systems and computer readable media are disclosed for a contextual image-based search. In one embodiment, a method includes receiving, at a computer implemented system for searching a knowledge base, an image submitted by a user of the system. The method continues with obtaining a classification label corresponding to the image, checking a context module database for the presence of a follow-up question associated with the classification label, and generating search term information for searching of the knowledge base, where the search term information comprises the classification label. If a follow-up question associated with the classification label is found, an embodiment further includes presenting the follow-up question to the user of the system, receiving from the user an answer to the follow-up question, modifying the search term information to include the answer, and checking the context module database for an additional follow-up question associated with the answer.

DETAILED DESCRIPTION

An embodiment of a method of performing an image-assisted search is illustrated by the screenshots shown inFIGS. 1A-1D. Screenshot100ofFIG. 1Adepicts a graphical user interface presented on a display screen of an information handling system of a user of an image-assisted search system. The information handling system may take any of the forms described above, including a personal computer or a mobile device. In the embodiment ofFIG. 1A, an icon105in the shape of a camera is presented for selection by the user if an image-assisted search is desired. Icon105is merely an example, and other user interface elements may be used to initiate an image-assisted search in alternative embodiments. A graphical user interface implementing an image-assisted search may include other information and user interface elements not shown in screenshot100.

Screenshot110ofFIG. 1Bdepicts a graphical user interface displayed at a time after a user has selected icon105of interface100to begin an image-assisted search process. In the embodiment ofFIG. 1B, a camera application window115has appeared, including a camera image window120and an image acquisition button125. In an embodiment, the user interface of screenshot110is displayed on a mobile device including a camera (such as a smartphone), and the image appearing in camera image window120is an image received by the camera of the mobile device. In an alternative embodiment, a user interface displayed on a device without a camera may include, for example, a file browser application window for selecting a stored image for upload instead of camera application window115.

Continuing with the example ofFIGS. 1A-1D, the image shown in image window120is submitted to a system as described herein for performing an image-assisted search. Screenshot130ofFIG. 1Cshows a pair of follow-up questions presented to the user by the system. At the time of screenshot130, the image-assisted search system has determined that the image submitted by the user depicts a Thunderbolt™ 3 USB Type-C port. The user interface depicted by screenshot130ofFIG. 1Cincludes questions135and140with respective radio buttons145and150for user response to the questions. In the embodiment ofFIG. 1C, selection of the radio button next to a question represents a “yes” answer to that question, and only one of the two radio buttons can be selected at a given time. In this example, the user has selected radio button150to indicate an interest in devices that fit into the Thunderbolt™ port, as opposed to systems including that port.

Screenshot155ofFIG. 1Ddepicts a graphical user interface provided in response to the user's answer to the questions shown in screenshot130ofFIG. 1C. In the embodiment ofFIG. 1D, the image-assisted search system has generated a search query and performed a search of a knowledge base using the query. Results of that search, including result items160,165,170and175, are displayed in the graphical user interface. In this embodiment, each result item includes an image and accompanying descriptive language. Either or both of the image and descriptive language may be configured as a hyperlink allowing access to additional information, in a manner that will be understood by those skilled in the art of computer-implemented search systems in view of this disclosure. In the example ofFIGS. 1A-1D, the knowledge base searched by the image-assisted search system is a database of available products. The knowledge base is not limited to this example, however, and could in other embodiments include technical support information, medical information, or any other information in a searchable data store.

An image-assisted search such as that illustrated inFIGS. 1A-1Dmay advantageously allow a user who does not know the name of a device or feature to nonetheless search for information regarding that device or feature. The methods and systems disclosed herein may also allow a user to determine a specific configuration or variety of a feature the user has only a general familiarity with. As another example in the area of computer components, a user may know that more memory for a particular computer system is needed, but not know the particular module that is compatible with the computer system. In an embodiment of an image-assisted search method, the user can submit a photograph of the memory module slots on the computer's motherboard. From this image the search system can determine the type of memory, such as double data rate fourth-generation dynamic random-access memory (DDR4 DRAM). The search system can then present a request to the user for identification of the microprocessor used by the computer. In an embodiment, the user submits a photograph of the computer's processor, from which the search system can identify the processor. The search system can then formulate a search query of its knowledge base for available compatible memory modules, perform the search, and present the results to the user. Further explanation and details are provided by the following embodiments.

A network environment200including multiple networked information handling systems is shown inFIG. 2. In the embodiment ofFIG. 2, client computing devices202(1)-(3), server computing device206, image-assisted search system208, server device206and server system210are communicatively coupled to one another via network204. Client computing devices202(1)-(3), server206and image-assisted search system208are embodiments of information handling systems as described above and elsewhere herein, and may take any of the various forms described, including personal computers, tablet computers, smartphones, or servers, as appropriate. As noted above, an information handling system may include an aggregate of instrumentalities. For example, as used in this disclosure, “server” may include a server system such as server system210, where a server system includes multiple networked servers configured for specific functions. As an example, server system210includes a messaging server212, web server214, application server216, database server218and directory server220, interconnected with one another via an intranet222. Network204includes one or more networks suitable for data transmission, which may include local area networks (LANs), wide area networks (WANs), storage area networks (SANs), the Internet, or combinations of these. In an embodiment, network106includes a publicly accessible network, such as a public switched telephone network (PSTN), a DSL connection, a cable modem connection or large bandwidth trunks (e.g., communications channels providing T1 or OC3 service). Such networks may also include cellular or mobile telephone networks and other wireless networks such as those compliant with the IEEE 802.11 standards. Intranet222is similar to network204except for being, typically, private to the enterprise operating server system210.

A block diagram illustrating certain components of an embodiment of image-assisted search system208is shown inFIG. 3. Although illustrated as a single device inFIG. 3, the image-assisted search system disclosed herein may also be implemented as a server system similar to server system210ofFIG. 2. In the embodiment ofFIG. 3, image-assisted search system208includes one or more network interfaces302, a processor304, memory306and data storage308. Memory306stores program instructions that when executed by processor304implement an image classification engine310, a context module312, a search query generator314and a search engine316. Data storage308is configured to store data including training images318, classification labels320, a context module database322and a knowledge base324.

Network interface302is configured for both sending and receiving of data and control information within a network. In an embodiment, network interface302comprises multiple interfaces and can accommodate multiple communications protocols and control protocols. Memory306includes a plurality of memory locations addressable by processor304for storing program instructions and data used in program execution. As such, memory306may be implemented using any combination of volatile or non-volatile memory, including random-access memory (RAM) and read-only memory (ROM). In an embodiment, memory306is system memory for processor304. Data storage308includes one or more integrated or peripheral mass storage devices, such as magnetic disks, optical disks, solid state drives or flash drives. In other embodiments, or at other times during operation of the embodiment ofFIG. 3, some or all of the instructions shown in memory306may be stored in data storage308, and some or all of the data shown in data storage308may be stored in memory306.

Image classification engine310is configured to classify a received image using a set of classification labels320, where the classification labels are relevant to the content of the knowledge base to be searched using image-assisted search system208. Image classification engine310implements a machine learning model, such as a convolutional neural network, that is trained using a set of training images318, where the training images are images similar to those expected to be submitted by users of image-assisted search system208. In a manner that will be understood by those of ordinary skill in the art of image classification in view of this disclosure, training of the machine learning model using training images318and classification labels320allows image classification engine310to classify subsequently-received images by assigning one of classification labels320to each received image. It is noted that while training images318are used during the training process for image classification engine310, training images318need not be retained after training when system208is used for classification of user-submitted images.

Context module312is configured to receive a classification label corresponding to an image submitted by a user, check context module database322to determine whether a follow-up question for the user is associated with the classification label, and generate search term information for searching of knowledge base324, where the search term information includes the received classification label. If there is a follow-up question associated with the classification label, the follow-up question is presented to the user, and an answer is received from the user. If there is then a follow-up question associated with the answer in the context module database, this question is also presented to the user, continuing until no additional follow-up questions are found. Search term information generated by context module312includes any answers received in addition to the classification label. In an embodiment, the search term information further includes field names from knowledge base324corresponding to the classification label and answers.

Search query generator314is configured to receive search term information generated by context module312and formulate a search query for searching knowledge base324. The search query includes the classification label and any answers included in the search term information, and is in the format of a query language compatible with knowledge base324. Search engine316applies the search query to knowledge base324and generates a list of search results for presentation to the user. In an embodiment, search engine316filters or ranks the search results for expected relevance, based on criteria such as popularity or user search history.

Knowledge base324is a data store searched by image-assisted search system208. In an embodiment, image classification engine310is trained to classify received images using classification labels relevant to the contents of knowledge base324, and context module database322includes follow-up questions designed to improve the relevance of search results obtained from knowledge base324. Knowledge base324may include any searchable information store, such as a database of product sales and/or support information, a store of academic research in a particular area, or the entire Internet.

Further alternatives and variations will be apparent to one of ordinary skill in the art in view of this disclosure. For example, some or all of the modules depicted within memory306may be implemented using separate servers as part of a server system like system210ofFIG. 2. Data depicted within data storage308may also be associated with one or more separate servers. Software modules and engines described herein may take various forms understood to one of ordinary skill in the art in view of this disclosure. A single module or engine described herein may in some embodiments be implemented by a combination of multiple files or programs. Alternatively or in addition, one or more functions associated with modules or engines delineated separately herein may be combined into a single file or program. As an example, search query generator314may in some embodiments be integrated with context module312or search engine316.

A process flow diagram illustrating the overall image-assisted search process is shown inFIG. 4. This type of diagram may also be referred to as an architecture diagram. In general, rectangular elements in the diagram ofFIG. 4refer to software engines or modules such as those depicted within memory306ofFIG. 3. Circular elements refer to inputs to or outputs from those modules or engines, and cylindrical elements represent data stores. Process flow400ofFIG. 4begins with a user405submitting a digital image410to the image-assisted search system. The image may be submitted by uploading via a web site, as an attachment to an email or text message, in an online chat session, through a smartphone application, through an augmented reality system such as one employing “smart glasses,” by insertion of a memory device into a kiosk system, or by any other method of transferring a digital image between a user and the image-assisted search system.

In some embodiments, the user's submission of image410includes a process of isolating a relevant region of the image. In the case of the image shown in camera image window120ofFIG. 1B, for example, time and computational resources can be saved if the part of the image addressed by the image classification engine is limited to the area around the computer port near the center of the image (or other region of interest, in the general case). Isolation of a relevant portion of the image may also be advantageous when an image contains multiple features that may correspond to classification labels. In an embodiment, a process of isolating a relevant region includes displaying the image to the user with an overlay delineating a proposed region (such as a bounding box) and receiving an indication from the user of whether the proposed region is the relevant region for the image-assisted search. In an embodiment for which a real-time image capture is used for the search (such as an application on a smartphone or augmented reality glasses), the overlay may be applied to the image in a camera image window such as window120ofFIG. 1B. Alternatively, a submitted image can be sent back to the user's device with an overlay applied. In a further embodiment, the overlay may be movable and/or size-adjustable by the user for delineation of the desired region, with the image and overlay then submitted to the image-assisted search system. In an embodiment, an initial proposed relevant region is determined using a separate algorithm from the algorithm used for image classification, such as an edge-detection algorithm.

When the image410is received by classification engine310(and any relevant-region isolation completed), the image is processed using the classification engine's trained image classification model to determine the best classification label415for the image. Training images318and classification labels320are shown as inputs to classification engine310, but it is noted that these inputs are provided during training of the image classification model, which happens before submission of image410. Training images318are images similar to those expected to be submitted by users searching knowledge base324. In an embodiment for which knowledge base324is a product database for a computer retailer, for example, training images318may include images of computers, computer ports, computer accessories, and portions of these things, where the images are taken from various directions and in various orientations.

In a further embodiment of process400not explicitly shown inFIG. 4, classification label415is presented to the user for verification once it has been determined by classification engine415. As an example, this verification step could take the form of a statement such as the “This looks like a Thunderbolt port” statement shown in screenshot130ofFIG. 1C, with an additional user response option along the lines of “Incorrect image identification.” In a further embodiment, classification engine310presents another possible classification label to the user in the event the user has indicated the original classification label is incorrect.

Context module312receives classification label415and searches context module database322to determine whether there is a follow-up question associated with classification label415. If a follow-up question is found, as illustrated by decision step420of process400, question425is presented to the user. In an embodiment, the question is displayed on the user's computing device via a graphical user interface, as shown in screenshot130ofFIG. 1C. Such a user interface may be presented as part of a web page or smartphone application, for example. In alternative embodiments, the follow-up question may be sent in written form by text message or email, or presented in spoken form using speech synthesis. An answer430from the user is received by context module312. Depending on the format of the question, the answer may take any of various forms, including but not limited to entered free text, text selected from a dropdown menu, a code representing a radio button or check box selection, or a transcribed voice response. Context module312again checks context module database322to determine whether an additional follow-up question is associated with answer430. If there is an additional follow-up question, the question and answer loop is repeated, until there is no follow-up question found for the last answer received.

When no follow-up question is found by searching the context module database, context module312provides search term information435to search query generator314. In the embodiment ofFIG. 4, search term information435includes one or more search terms and one or more knowledge base field names. The search terms in search term information435include the classification label415and any answers430received by the context module. One or more of classification label415and any answers430may also be associated with a knowledge base field name in context module database322. Associating a knowledge base field name with a search term allows a search query to be generated that instructs the search engine to search for a particular term within a specific field of a database representing the knowledge base. This localization of the search can improve the relevance of the search results. In an alternative embodiment, search term information435does not include knowledge base field names. Such an embodiment can be used if the knowledge base is not organized in a way that uses fields, or if searching across all fields of the knowledge base is acceptable. An example of how knowledge base fields may be associated with classification labels or answers in context module database322is discussed below in connection withFIGS. 6A-6D.

Search query generator314converts search term information435into a search query440compatible with search engine316and knowledge base324. For example, if knowledge base324is a database designed to use structured query language (SQL), search query440may be in the form of one or more SQL queries. Search engine316applies search query440to knowledge base324and obtains a list of search results445which are presented to the user.

A simplified example of a table within a knowledge base to be searched is shown inFIG. 5. Product table500ofFIG. 5is a table of product information that might be maintained by a computer retailer. Table500has been made unrealistically small for purposes of explanation, and is not intended to completely (or necessarily accurately) describe any particular products mentioned therein. In an embodiment, entries in table500are linked (for example, via a Universal Research Locator (URL) for the World Wide Web) to more extensive information, such as data sheets, for the products described by the entries. The information in table500is organized using fields510,520and530, which have respective field names of “Product Name,” “Product Type” and “Tech Specs.”

Even from the simplified example of product table500, it can be seen that a product database for a technological product can reflect an extensive array of detailed options. Taking the example of computer interfaces, three different port types (Thunderbolt, HDMI, USB) can be seen in Table500alone, and a complete computer product table would include several others as well. A non-enthusiast computer user could easily be unaware of the names of all of the ports, and might therefore not have a place to begin searching a product database for an accessory compatible with a particular port observed on their own computer. The image-assisted search disclosed herein can allow a user to initiate a search by submitting a photograph of such a port. Even when the name of a feature is known, that name may appear in multiple different contexts in a knowledge base. For example, “Thunderbolt” appears in six of the entries in table500. Some of those entries are for accessories that plug into a Thunderbolt™ port, while others are for systems that have a Thunderbolt™ port. A simple search of table500for “Thunderbolt” will therefore return a long list of results, many of which may not be relevant to the user. The context module of the search disclosed herein can provide search queries that improve the relevance of search results by narrowing the scope of a search.

A database schema diagram illustrating the structure of one embodiment of context module database322is shown inFIG. 6. Schema diagram600includes table block605representing a follow-up question table, table block610representing a classification label table and table block615representing an answer table. The follow-up question table referenced in block605includes five fields: Classification Label ID, Follow-up Question ID, Question Text, Answer ID and Question Type. The classification label table referenced in block610includes three fields: Classification Label ID, Classification Label and Knowledge Base Field Name. The answer table referenced in block615includes four fields: Answer ID, Answer, Recursive Follow-up Question ID, and Knowledge Base Field Name. The follow-up question table is connected to the classification label table via the Classification Label ID field common to both tables. The follow-up question table is connected to the answer table via the Answer ID field common to both tables. The Classification Label ID field is a primary key to the classification label table, and the Answer ID field is a primary key to the answer table.

Illustrative portions of the tables referenced in the diagram ofFIG. 6Aare shown inFIGS. 6B-6D. Portion620of a follow-up question table is shown inFIG. 6B, portion640of a classification label table is shown inFIG. 6C, and portion660of an answer table is shown inFIG. 6D. For clarity of explanation, table portion620ofFIG. 6Bis limited to entries associated with a single classification label ID of “1234”. A full follow-up question table would include entries for any classification labels that follow-up questions have been generated for. The relationship between classification label IDs, such as ID 1234 of table620, and plain-text classification labels is stored in the classification label table ofFIG. 6C. As shown in table portion640ofFIG. 6C, classification label ID 1234 corresponds to the classification label “Thunderbolt port.” In the embodiment ofFIGS. 6A-6D, a numeric classification label identifier (ID) is associated with each plain-text classification label. The classification label ID can be used to represent the classification label for purposes internal to the image-assisted search system, and converted to the classification label for situations in which plain text is needed, such as presentation to a user or text-based searching. For purposes of this disclosure, “classification label” can refer to either a plain-text label or to an identifier representing that label. Therefore, for example, transmission of classification label415from classification engine310to context module312as shown inFIG. 4may be implemented by transmission of a classification label ID associated with classification label415.

Returning toFIG. 6B, the four entries in follow-up question table portion620reflect four follow-up questions associated with the Thunderbolt port classification label ID. Each of the follow-up questions is assigned a follow-up question ID, an answer ID, a question type, and a string of question text. The follow-up question ID uniquely identifies a combination of a classification label ID and follow-up question, and can be used in the answer table ofFIG. 6Dto identify “recursive follow-up questions,” or additional follow-up questions associated with answers to previous follow-up questions. An answer ID in the third column of table portion620is a numeric identifier representing an answer to the question associated with the follow-up question ID. The relationship between answer IDs and plain-text answers is stored in the answer table ofFIG. 6D. For purposes of this disclosure, “answer” can refer to either a plain-text answer or to an identifier representing that answer. Therefore, for example, submission of answer430to context module312as shown inFIG. 4may be implemented by transmission from the user's device to the context module of an answer ID associated with the answer.

The question type field of table portion620inFIG. 6Bidentifies the format of the question associated with the follow-up question ID. In addition to the radio button and check box question types shown in table portion620, other question types may be used as appropriate, including dropdown menus, requests for free text response, and requests for submission of an additional image. Questions and answers may be submitted via a graphical user interface or in other ways such as by voice synthesis and voice response, by text message or online chat, or by email. In an embodiment, the question type in table portion620may be represented by a code rather than a plain-text name. The question text field of table portion620includes the text content of the question, which may be combined with answers from answer table portion660when presenting possible answers for selection by the user.

Moving toFIG. 6C, classification label table portion640relates classification label IDs to plain-text classification labels, as noted above. Table portion640also relates a knowledge base field name to each classification label ID. In an embodiment, the knowledge base field name associated with a classification label is the name of a field in the knowledge base that should be searched using the classification label as a search term. For example, the knowledge base field name column of table portion640indicates that searches for “Thunderbolt port” or “USB port” should be carried out in the Tech Specs field of product table500inFIG. 5(the knowledge base used for this example). By contrast, searches for “mouse” or “keyboard” should be carried out in the Product Name field of product table500. In some embodiments, multiple knowledge base field names can be included in a knowledge base field name field within the context module database.

Answer table portion660inFIG. 6Drelates answer IDs to plain-text answers, as noted above in the discussion ofFIG. 6B. In the Recursive Follow-up Question ID field, follow-up question IDs are associated with some of the answers to previously-asked questions. For example, the first entry of table portion660indicates that if answer ID “1” is received as a response to follow-up question “abc” in table portion620, additional follow-up question “ghi” is then presented to the user. Table portion660also relates a knowledge base field name to each answer ID. In an embodiment, the knowledge base field name associated with an answer is the name of a field in the knowledge base that should be searched using the answer as a search term. For example, the knowledge base field name column of table portion660indicates that searches for “Systems,” “Accessories,” “Laptops,” “Tablets” or “Desktops” should be carried out in the Product Type field of product table500inFIG. 5. By contrast, searches for “Ethernet port,” “HDMI port” or “VGA port” should be carried out in the Tech Specs field of product table500.

Use of the context module database embodiment ofFIGS. 6A-6Dcan be illustrated by an example in which the classification label “Thunderbolt port” (or corresponding classification label ID “1234”) is received by context module312. Turning to follow-up question table portion620, there are four follow-up question IDs associated with the received classification label ID. A method of selecting the appropriate questions to present to the user is described by the following pseudocode for a first call to the context module database:

Select all from Follow-up Question table

where Classification Label ID equals myClassificationLabelID

and Follow-up Question ID does not equal Answer table's Recursive Follow-up Question ID

join Answer table on Answer ID

In the pseudocode above, the classification label ID received from classification engine310is assigned to the variable “myClassificationLabelID.” The method implemented by the pseudocode selects all follow-up questions in the follow-up question table620having a classification label ID matching the received classification label ID, but removes those questions having a follow-up question ID matching a recursive follow-up question ID in answer table660. The “does not equal” line of the pseudocode removes follow-up question IDs “ghi” and “jkl” from the selected questions, since those follow-up question IDs appear in the Recursive Follow-up Question ID field of answer table660. The selected questions are therefore the questions identified by follow-up question IDs “abc” and “def”. Because the question type for these questions is designated as a radio button in table portion620, a pair of radio buttons associated with the corresponding text from the Question Text field is presented to the user. The result is a pair of questions similar to those shown in screenshot130ofFIG. 1C.

When an answer to the presented question is received, answer table portion660is checked to determine whether an additional follow-up question is associated with the received answer. For example, if the user chooses answer ID “2” rather than answer ID “1” in response to the presented radio buttons (i.e., selecting devices using the port rather than systems having the port), no follow-up question ID corresponding to an answer ID of 2 is shown in the Recursive Follow-up Question ID field of table660. There are therefore no additional follow-up questions to present to the user. Context module312generates search term information for search query generator14. In the current example, the search terms within the search term information include the classification label “Thunderbolt port” and the answer “Accessories” (corresponding to answer ID “2”). In an embodiment, the search term information also includes the knowledge base field names associated with the search terms. In such an embodiment, the search term information would indicate that “Thunderbolt port” should be searched in the knowledge base field Tech Specs, and “Accessories” should be searched in the knowledge base field Product Type. Generation of an appropriate search query by search query generator314and application of that search query to product table500would produce a set of results including the second, third, fifth and sixth entries in table500. A list of these results would then be presented to the user.

As an “alternative ending” to the above example, consider that the same question is presented to the user, but the user chooses answer ID “1” in response rather than answer ID “2”. In this case, a follow-up question ID of “ghi” corresponding to an answer ID of “1” is shown in the Recursive Follow-up Question ID field of answer table660. If the follow-up question ID “ghi” determined from table660is assigned to a variable called myRecursiveFollowupQuestionID, selection of the next follow-up question to present to the user is described by the following pseudocode:

Select all from Follow-up Question table

where Classification Label ID equals myClassificationLabelID

and Follow-up Question ID equals myRecursiveFollowupQuestionID

join Answer table on Answer ID

The method implemented by the above pseudocode simply selects the follow-up question ID associated with the received answer in answer table660. Follow-up question table620shows that selected follow-up question ID “ghi” corresponds to a checkbox having as possible answers the answers associated with answer IDs 3, 4 and 5. In the embodiment ofFIGS. 6A-6D, context module312employs a rule that for checkbox questions such as question “ghi”, the question text from table620is combined with the plain-text answers from table660to produce a question of the form:

Would you like to look at:LaptopsTabletsDesktops

An answer to the above follow-up question can include any or all of the three checkbox choices. Question “ghi” is presented to the user. Assuming at least one of the boxes is checked by the user, answer table660indicates that another follow-up question, having follow-up question ID “jkl” is associated with the answer. Follow-up question table620shows that question “jkl” is a checkbox question about additional ports, with possible answers corresponding to answer IDs 6, 7 and 8. Question “jkl” is presented to the user. Because none of answer IDs 6, 7 or 8 is associated with an additional follow-up question ID in answer table660, the recursive question process ends at that point. In an embodiment where the user selects “Laptops” in response to question “ghi” and “HDMI port” in response to question “jkl”, search terms in the search term information435generated by context module312for this example include classification label “Thunderbolt port” and answers “Systems,” “Laptops,” and “HDMI port.” In a further embodiment for which knowledge base field names are included in search term information435, the search term information indicates that “Thunderbolt port” and “HDMI port” should be searched in the Tech Specs field of product table500, and that “Systems” and “Laptops” should be searched in the Product Type field of product table500.

The database schema and tables shown inFIGS. 6A-6Drepresent one implementation of a context module database as described herein. Multiple alternatives and variations will be apparent to one of ordinary skill in the art of database design in view of this disclosure.

A flow chart illustrating an embodiment of a method for performing an image-assisted search is shown inFIG. 7. Method700ofFIG. 7is an embodiment of a method performed by an image-assisted search system such as system208ofFIG. 3. In some embodiments, before method700begins, the image-assisted search system is pre-configured for searching of a target knowledge base. The pre-configuration includes training the machine learning model employed by the image classification engine within the search system. In an embodiment, the training uses a set of training images similar to those expected to be submitted by users, and trains the model to classify the training images using a set of classification labels expected to be useful as search terms for searching the knowledge base. Pre-configuration of the image-assisted search system also includes populating a context module database, such as database322ofFIGS. 3 and 4, with initial follow-up questions associated with some or all of the classification labels to be used by the image classification engine, as appropriate. In an embodiment, populating of the context module database also includes associating additional follow-up questions with some of the answers to be received in response to the initial follow-up questions. In some embodiments, populating of the context module database includes associating field names from the target knowledge base with classification labels and with answers.

Search method700begins at step705with receiving an image from a user of the system. The image can be received in various ways discussed further above in connection with the process flow ofFIG. 4. The method continues in step710with obtaining a classification label corresponding to the image. The classification label is determined by a classification engine within the search system, as discussed further above in connection withFIGS. 3 and 4. In step715of method700, a contest module database is checked for one or more follow-up questions associated with the classification label. If no follow-up question is found (“N” branch of decision step720), the method continues with generating a knowledge base search query containing the classification label (step725). In an embodiment, the knowledge base search query is generated by a search query generator module of the search system, based on search term information generated by a context module of the system. The generated search query may in some embodiments contain a field name from the knowledge base to be searched in addition to the classification label. This branch of the method continues with performing a search of the knowledge base using the search query, and presenting results of the search to the user.

If a follow-up question associated with the classification label is found (“Y” branch of decision step720), the question is presented to the user (step740) and an answer to the question is received (step745). The question and answer may be transmitted in various ways, as discussed further above in connection withFIG. 4. The method continues with checking the context engine database for an additional follow-up question associated with the received answer. If an additional follow-up question associated with the received answer is found (“Y” branch of decision step755), the loop including steps740through755is repeated until no additional follow-up question is found. When no additional follow-up question is found (“N” branch of decision step755), the method continues with generating a knowledge base search query containing the classification label and any answers received to follow-up questions (step760). In an embodiment, the search query contains one or more field names from the knowledge base in addition to the classification label and answers. The method continues with performing a search of the knowledge base using the search query, and presenting results of the search to the user.

FIG. 8depicts a block diagram of an information handling system810suitable for implementing aspects of the systems described herein. In the embodiment ofFIG. 8, computing system810implements image-assisted search system208. Embodiments of the computing system ofFIG. 8can, alternatively or in addition, implement various other engines and modules described in this disclosure. Computing system810broadly represents any single or multi-processor computing device or system capable of executing computer-readable instructions. Examples of computing system810include, without limitation, any one or more of a variety of devices including workstations, personal computers, laptops, client-side terminals, servers, distributed computing systems, handheld devices (e.g., personal digital assistants and mobile phones), network appliances, switches, routers, storage controllers (e.g., array controllers, tape drive controller, or hard drive controller), and the like. In its most basic configuration, computing system810may include at least one processor814and a system memory816. By executing the software that implements image-assisted search system208, computing system810becomes a special purpose computing device that is configured to perform contextual, image-assisted generation of search term information in manners described elsewhere in this disclosure.

Processor814generally represents any type or form of processing unit capable of processing data or interpreting and executing instructions. In certain embodiments, processor814may receive instructions from a software application or module. These instructions may cause processor814to perform the functions of one or more of the embodiments described and/or illustrated herein. System memory816generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and/or other computer-readable instructions. Examples of system memory816include, without limitation, random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory device. The ROM or flash memory can contain, among other code, the Basic Input-Output System (BIOS) which controls basic hardware operation such as the interaction with peripheral components. Although not required, in certain embodiments computing system810may include both a volatile memory unit (such as, for example, system memory816) and a non-volatile storage device (such as, for example, primary storage device832, as described further below). In one example, program instructions executable to implement a forwarding engine configured to forward messages using bit indexed explicit replication (BIER) may be loaded into system memory816.

In certain embodiments, computing system810may also include one or more components or elements in addition to processor814and system memory816. For example, as illustrated inFIG. 8, computing system810may include a memory controller818, an Input/Output (I/O) controller820, and a communication interface822, each of which may be interconnected via a communication infrastructure812. Communication infrastructure812generally represents any type or form of infrastructure capable of facilitating communication between one or more components of a computing device. Examples of communication infrastructure812include, without limitation, a communication bus (such as an Industry Standard Architecture (ISA), Peripheral Component Interconnect (PCI), PCI express (PCIe), or similar bus) and a network.

Memory controller818generally represents any type or form of device capable of handling memory or data or controlling communication between one or more components of computing system810. For example, in certain embodiments memory controller818may control communication between processor814, system memory816, and I/O controller820via communication infrastructure812. In certain embodiments, memory controller818may perform and/or be a means for performing, either alone or in combination with other elements, one or more of the operations or features described and/or illustrated herein. I/O controller820generally represents any type or form of module capable of coordinating and/or controlling the input and output functions of a computing device. For example, in certain embodiments I/O controller820may control or facilitate transfer of data between one or more elements of computing system810, such as processor814, system memory816, communication interface822, display adapter826, input interface830, and storage interface834.

Communication interface822broadly represents any type or form of communication device or adapter capable of facilitating communication between computing system810and one or more additional devices. For example, in certain embodiments communication interface822may facilitate communication between computing system810and a private or public network including additional computing systems. Examples of communication interface822include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, and any other suitable interface. In at least one embodiment, communication interface822may provide a direct connection to a remote server via a direct link to a network, such as the Internet. Communication interface822may also indirectly provide such a connection through, for example, a local area network (such as an Ethernet network), a personal area network, a telephone or cable network, a cellular telephone connection, a satellite data connection, or any other suitable connection.

In certain embodiments, communication interface822may also represent a host adapter configured to facilitate communication between computing system810and one or more additional network or storage devices via an external bus or communications channel. Examples of host adapters include, without limitation, Small Computer System Interface (SCSI) host adapters, Universal Serial Bus (USB) host adapters, Institute of Electrical and Electronics Engineers (IEEE) 11054 host adapters, Serial Advanced Technology Attachment (SATA) and external SATA (eSATA) host adapters, Advanced Technology Attachment (ATA) and Parallel ATA (PATA) host adapters, Fibre Channel interface adapters, Ethernet adapters, or the like. Communication interface822may also allow computing system810to engage in distributed or remote computing. For example, communication interface822may receive instructions from a remote device or send instructions to a remote device for execution.

As illustrated inFIG. 8, computing system810may also include at least one display device824coupled to communication infrastructure812via a display adapter826. Display device824generally represents any type or form of device capable of visually displaying information forwarded by display adapter826. Similarly, display adapter826generally represents any type or form of device configured to forward graphics, text, and other data from communication infrastructure812(or from a frame buffer) for display on display device824. Computing system810may also include at least one input device828coupled to communication infrastructure812via an input interface830. Input device828generally represents any type or form of input device capable of providing input, either computer or human generated, to computing system810. Examples of input device828include, without limitation, a keyboard, a pointing device, a speech recognition device, or any other input device.

As illustrated inFIG. 8, computing system810may also include a primary storage device832and a backup storage device833coupled to communication infrastructure812via a storage interface834. Storage devices832and833generally represent any type or form of storage device or medium capable of storing data and/or other computer-readable instructions. For example, storage devices832and833may include a magnetic disk drive (e.g., a so-called hard drive), a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash drive, or the like. Storage interface834generally represents any type or form of interface or device for transferring data between storage devices832and833and other components of computing system810. A storage device like primary storage device832can store information such as routing tables and forwarding tables.

In certain embodiments, storage devices832and833may be configured to read from and/or write to a removable storage unit configured to store computer software, data, or other computer-readable information. Examples of suitable removable storage units include, without limitation, a floppy disk, a magnetic tape, an optical disk, a flash memory device, or the like. Storage devices832and833may also include other similar structures or devices for allowing computer software, data, or other computer-readable instructions to be loaded into computing system810. For example, storage devices832and833may be configured to read and write software, data, or other computer-readable information. Storage devices832and833may be a part of computing system810or may in some embodiments be separate devices accessed through other interface systems. Many other devices or subsystems may be connected to computing system810. Conversely, all of the components and devices illustrated inFIG. 8need not be present to practice the embodiments described and/or illustrated herein. The devices and subsystems referenced above may also be interconnected in different ways from that shown inFIG. 8.

Computing system810may also employ any number of software, firmware, and/or hardware configurations. For example, one or more of the embodiments disclosed herein may be encoded as a computer program (also referred to as computer software, software applications, computer-readable instructions, or computer control logic) on a computer-readable storage medium. Examples of computer-readable storage media include magnetic-storage media (e.g., hard disk drives and floppy disks), optical-storage media (e.g., CD- or DVD-ROMs), electronic-storage media (e.g., solid-state drives and flash media), and the like. Such computer programs can also be transferred to computing system810for storage in memory via a network such as the Internet or upon a carrier medium. The computer-readable medium containing the computer program may be loaded into computing system810. All or a portion of the computer program stored on the computer-readable medium may then be stored in system memory816and/or various portions of storage devices832and833. When executed by processor814, a computer program loaded into computing system810may cause processor814to perform and/or be a means for performing the functions of one or more of the embodiments described and/or illustrated herein. Additionally or alternatively, one or more of the embodiments described and/or illustrated herein may be implemented in firmware and/or hardware. For example, computing system810may be configured as an application specific integrated circuit (ASIC) adapted to implement one or more of the embodiments disclosed herein.

The above-discussed embodiments can be implemented by software modules that perform one or more tasks associated with the embodiments. The software modules discussed herein may include script, batch, or other executable files. The software modules may be stored on a machine-readable or computer-readable storage media such as magnetic floppy disks, hard disks, semiconductor memory (e.g., RAM, ROM, and flash-type media), optical discs (e.g., CD-ROMs, CD-Rs, and DVDs), or other types of memory modules. A storage device used for storing firmware or hardware modules in accordance with an embodiment can also include a semiconductor-based memory, which may be permanently, removably or remotely coupled to a microprocessor/memory system. Thus, the modules can be stored within a computer system memory to configure the computer system to perform the functions of the module. Other new and various types of computer-readable storage media may be used to store the modules discussed herein.