Searching for similarities: from interior design to medical diagnostics
By: Bartek Borucki
Tags: Search, Machine Learning, Radiology, AI, Multimodality

Have you ever wondered what interior design has in common with medical diagnostics? No? Really? So, how about this joke:

Q: What is the difference between interior design and medical diagnostics?

A: None! From the perspective of Artificial Intelligence systems.

Read on and I’ll reveal why.

AI revolution…

Source: Laurent / stock.adobe.com

Now, let’s be serious for a while. Artificial Intelligence (AI) is already revolutionizing our day-to-day life. June Oven uses built-in cameras to watch our food and the preparation process to help us master cooking. Voyage enhances elderly residents’ life quality in Florida with door-to-door self-driving cars, monitoring also their emotions [1]. Arterys is one of the first to focus on medical imaging to help doctors with better and faster diagnostics with first AI-based FDA clearance for blood flow analysis, and is now spreading to liver and lung lesions with their cloud image processing platform. As we are still very far away from artificial general intelligence, today’s use cases are based on the so-called narrow AI — a model that is designed and tuned for a specific single task, e.g. pulmonary nodules detection in chest X-Ray. Narrow AI is still a capacious term and a definition is broad though and this article is not the best place to discuss it. In fact, the majority of AI solutions nowadays are based on the concepts of machine learning (ML) and, particularly, deep learning (DL), especially, where images or texts are the information medium. In its most basic definition machine learning is the capability of computer systems to learn from data without human intervention, while deep learning uses complex multi-layered neural networks to perform this task. What is most interesting though is that AI already beats human performance in a number of such tasks, as more clues and evidence can be deeply observed.

Under the hood of Watson

You all probably have heard of IBM Watson. In 2011 a computer system designed and built by IBM competed in Jeopardy! quiz show against its two human champions, outplaying them both. The system answered a variety of natural language questions using only its internal algorithms based on data input available from multiple sources, including encyclopedias, dictionaries, thesauri, news articles and literary works, also with other databases, taxonomies and ontologies. What is under the hood then? Put simply, a complex search engine. Watson’s algorithms contextually transform the question into an abstract query, retrieve the related content from documents and repositories, extract possible words and initially score them as answers, to filter and match them with more related content. In the final step, Watson’s algorithms produce a confidence ranking by many independent algorithms that are trying to determine which of the retrieved pieces of evidence support the candidate answers. IBM named it DeepQA engine and what is most important inside is that the evidence is evaluated within temporal reasoning, geospatial reasoning and statistical paraphrasing (or mapping), and in the crucial phase it is, in fact, a combination of a number of algorithms designed to work with different types of data, e.g. texts or images.

If you think of these components in the medical context, temporal reasoning becomes a patient history or sequence of symptoms, geospatial reasoning becomes human body mapping, while statistical paraphrasing covers domain-specific terminology. So Watson should perfectly fit into the problems of a medical diagnostics. And in fact, it does. So far Watson Health has been tested in a number of healthcare-related tasks, specifically in cancer diagnosis and treatment. Still, its power lies in the combination of common analysis of the numeric findings (i.e. diagnostic examinations), textual descriptions of the symptoms, medical imaging, patient history, and the knowledge base, both in the hospital assets and in the medical publications. Is it already successful? It is still controversial, but hey, we are on the bleeding edge of AI technology and what is most important here is that it is the very variety of data sources analysed in combination that drives this approach.

Searching with multimodality

So let’s generalize the concept above and then map it back to practice. In fact, we have just three fundamental steps:

The problem description is translated into a structured language, which is understandable for the analytic algorithms. Multiple data sources are blended into a consistent representation. Such representations can be matched and scored — quantitatively compared to each other.

Source: maxsim / stock.adobe.com

Now imagine you can do it a number of times with different data representations, multiple data combinations and multiple candidate matching results. This is no different than a concept of a multimodal search engine. And the very multimodality here is understood as a combination of various data sources in a single analytical task — be it a comparison or search task.

What is the practical problem? Let’s focus on the medical diagnostics again. The holy grail of the automated diagnostics is, of course, the very Watson-like combination — understand and merge multiple types, forms and sources of patient data, compare it with historical cases and domain knowledge or reference data, find the best-fitted treatment and optimize outcomes. But there’s a huge number of intermediate solutions that are still important steps towards healthcare revolution. Take an example of the first patient-doctor contact — put it in a booth, provide a few basic symptoms, obtain basic data like ECG or pulmonary sounds, and you get an AI-based automated, fast and cheap solution of the initial diagnosis. Take an examination referral plus the outcome image (e.g. MRI or CAT scan) and compare it to similar historical cases to generate the personalized template of study description. Use the structured descriptors of a medical case to sort the medical cases based on the severity of the automatically-found diseases or its progress degree. What you get in this case is an optimized workflow for radiologists or the ER. And so on, and forth.

So, what is with this interior design anyway?

Source: onzon / stock.adobe.com

Don’t worry, we are almost there. In 2017, as the answer to “What Looks Good with my Sofa?”, Tooploox proposed the concept of the AI-based multi-modal search engine for interior design that combines visual and textual queries, called StyleSearch [2]. The goal of this engine was to retrieve interior objects, e.g. furniture or decorations, that share visual and aesthetic similarities with the query. This search engine allows the user to take a photo of a room and retrieve, with a high recall, a list of items that are visually identical or similar to those present in the photo, or return other items that aesthetically and stylistically match together. Additionally, it allows you to extend the query with textual description providing modifications or clarifications of search parameters. For example, one can provide a photo of the red chair and description “in fluffy black, with armrests” to search for a similar product but with given criteria. The engine is based on a number of text and image processing techniques, but its heart is driven by AI core. The breakthrough idea behind this solution is the concept of the stylistic match and the multimodality of both source and target information.

In StyleSearch engine room images are pre-processed by using a golden standard object detection algorithms, such as YOLO 9000, to obtain a higher precision compared to whole image analysis and work without prior object categorization (which is a limitation for a majority of visual search methods). Multiple neural networks which are pre-trained on ImageNet translate images to vector representations. CBOW algorithm is used to translate contextual domain-specific information — here, the furniture and interior, while the inter-object relations are mapped in the feature space, e.g. to find objects present in the same room. NLP model (LSTM deep neural network) maps the textual query to object embedding space. Finally, the algorithm blends multiple modalities in the manner similar to DeepQA model — by re-ranking the results of the initial searches by the distance between the query and the item in visual feature space for each modality independently. The final result is a set of closest matches. Additionally, the style similarity measure is also proposed in the paper [2].

In the majority of common search engines nowadays you can just use a single data type as a description of your query. Most often you type in some text (like in Google search) or provide just a single image (like in Google Goggles or Pinterest visual search). But the possibilities of advanced searching and matching applications open with the multimodal approach. And the key is again in the deep combination of the modalities in contrast to simple parallel usage of multiple uni-modal searches. In such an approach both the search description and the compared data can be of multiple types, formats, semantics, content, and so on. Moreover, you don’t have to manually tag the searched content anymore, as it is also automatically translated into a query-context-related description.

In other words, the multimodal search engine used in the StyleSearch solution maps the problem from various sources of data to a structured descriptor, collects multiple evidence and scores matches to choose the best answer. Sounds familiar? Indeed, it seems like the process of interior design i.e. the stylistic matching of multiple entities resembles the process of medical diagnostics i.e. the matching of symptoms, medicines, imaging, illness history, diagnosis, treatment, etc. There is hardly any difference between the two from the perspective of the AI-based decision support system. One can argue of course that differentiation between cat and dog or fluffy sofa and leather sofa is an order of magnitude simpler than differentiation of medical cases, but… Whether it is Watson playing Jeopardy!, or StyleSearch looking for your new sofa, or a radiologist describing the clinical case, or medical doctor looking for the optimal diagnosis and treatment for his patient — they all fall into in-the-brain or on-the-chip multimodal matching engine. In fact, it is a next small step towards widening the narrow AI from single task & single source to single task & multiple sources, and ultimately to multiple tasks.

On the road to revolutionize healthcare?

Source: phonlamaiphoto / stock.adobe.com

Tooploox is already on track to make a leap towards modification of multimodal search solution to match the data and the requirements of medical applications and to validate the method applicability — particularly within medical imaging. Multiple data sources can also nicely map to several imaging modalities or be combined with a variety of forms of medical datasets. This will be the next step on their AI journey within healthcare and well-being. And frankly speaking, we can expect impressive results looking back at previous works by Tomasz Trzciński et consortes — starting with recent MICCAI-presented work on the monitoring of Achilles tendon healing process (collaboration of my group at the Interdisciplinary Centre for Mathematical and Computational Modelling with Tomasz and his Tooploox team) [3], through vessel segmentation in CT scans, to using Generative Adversarial Networks for better biometric data privacy [4].

At the same time, in the academic background of Tooploox team, the collaborative initiatives are focused on the accompanying research to complement this landscape with more practical tools covering explainable AI in the context of opening the black-box of deep neural networks and understanding the decision processes of AI, the influence of data reduction on the decision systems in the context of medical data privacy and anonymization, and finally, the problems of knowledge exchange between multiple deep learning models to make the next step towards even more generic AI.

Can you see more interesting healthcare and well-being applications? Or maybe this solves the problem you’re struggling with in your computer-assisted medicine applications? Feel free to discuss below or just reach out to us on www or social media.