abstract: Machine Learning has been a big success story during the AI resurgence. One particular stand out success relates to learning from a massive amount of data. In spite of early assertions of the unreasonable effectiveness of data, there is increasing recognition for utilizing knowledge whenever it is available or can be created purposefully. In this paper, we discuss the indispensable role of knowledge for deeper understanding of content where (i) large amounts of training data are unavailable, (ii) the objects to be recognized are complex, (e.g., implicit entities and highly subjective content), and (iii) applications need to use complementary or related data in multiple modalities/media. What brings us to the cusp of rapid progress is our ability to (a) create relevant and reliable knowledge and (b) carefully exploit knowledge to enhance ML/NLP techniques. Using diverse examples, we seek to foretell unprecedented progress in our ability for deeper understanding and exploitation of multimodal data and continued incorporation of knowledge in learning techniques.
  .
  .
  .
  Machine Intelligence, Multimodal Exploitation, Understanding Complex Text, Knowledge-enhanced Machine Learning, Knowledge-enhanced NLP, Knowledge-driven Deep Content Understanding, Personalized Digital Health, Semantic-Cognitive-Perceptual Computing, Implicit Entity Recognition, Emoji Sense Disambiguation
author: Amit Sheth; Sujan Perera; Sanjaya Wijeratne; Krishnaprasad Thirunarayan
title: Knowledge will Propel Machine Understanding of Content: Extrapolating from Current Examples

# Introduction

Recent success in the area of Machine Learning (ML) for Natural Language Processing (NLP) has been largely credited to the availability of enormous training datasets and computing power to train complex computational models . Complex NLP tasks such as statistical machine translation and speech recognition have greatly benefited from the Web-scale unlabeled data that is freely available for consumption by learning systems such as deep neural nets. However, many traditional research problems related to NLP, such as part-of-speech tagging and named entity recognition (NER), require labeled or human-annotated data, but the creation of such datasets is expensive in terms of the human effort required. In spite of early assertion of the unreasonable effectiveness of data (i.e., data alone is sufficient), there is an increasing recognition for utilizing knowledge to solve complex AI problems. Even though knowledge base creation and curation is non-trivial, it can significantly improve result quality, reliability, and coverage. A number of AI experts, including Yoav Shoham , Oren Etzioni, and Pedro Domingos , have talked about this in recent years. In fact, codification and exploitation of declarative knowledge can be both feasible and beneficial in situations where there is not enough data or adequate methodology to learn the nuances associated with the concepts and their relationships.

The value of domain/world knowledge in solving complex problems was recognized much earlier . These efforts were centered around language understanding. Hence, the major focus was towards representing linguistic knowledge. The most popular artifacts of these efforts are FrameNet  and WordNet , which were developed by realizing the ideas of frame semantics  and lexical-semantic relations , respectively. Both these resources have been used extensively by the NLP research community to understand the semantics of natural language documents.

The building and utilization of the knowledge bases took a major leap with the advent of the Semantic Web in the early 2000s. For example, it was the key to the first patent on Semantic Web and a commercial semantic search/browsing and personalization engine over 15 years ago , where knowledge in multiple domains complemented ML techniques for information extraction (NER, semantic annotation) and building intelligent applications[^1]. Major efforts in the Semantic Web community have produced large, cross-domain (e.g., DBpedia, Yago, Freebase, Google Knowledge Graph) and domain specific (e.g., Gene Ontology, MusicBrainz, UMLS) knowledge bases in recent years which have served as the foundation for the intelligent applications discussed next.

The value of these knowledge bases has been demonstrated for determining semantic similarity , question answering , ontology alignment , and word sense disambiguation (WSD) , as well as major practical AI services, including Apple's Siri, Google's Semantic Search, and IBM's Watson. For example, Siri relies on knowledge extracted from reputed online resources to answer queries on restaurant searches, movie suggestions, nearby events, etc. In fact, "question answering", which is the core competency of Siri, was built by partnering with Semantic Web and Semantic Search service providers who extensively utilize knowledge bases in their applications[^2]. The Jeopardy version of IBM Watson uses semi-structured and structured knowledge bases such as DBpedia, Yago, and WordNet to strengthen the evidence and answer sources to fuel its DeepQA architecture . A recent study  has shown that Google search results can be negatively affected when it does not have access to Wikipedia. Google Semantic Search is fueled by Google Knowledge Graph[^3], which is also used to enrich search results similar to what the Taalee/Semagix semantic search engine did 15 years ago[^4] .

While knowledge bases are used in an auxiliary manner in the above scenarios, we argue that they have a major role to play in understanding real-world data. Real-world data has a greater complexity that has yet to be fully appreciated and supported by automated systems. This complexity emerges from various dimensions. Human communication has added many constructs to language that help people better organize knowledge and communicate effectively and concisely. However, current information extraction solutions fall short in processing several implicit constructs and information that is readily accessible to humans. One source of such complexity is our ability to express ideas, facts, and opinions in an implicit manner. For example, the sentence *"The patient showed accumulation of fluid in his extremities, but respirations were unlabored and there were no use of accessory muscles"* refers to the clinical conditions of "shortness of breath" and "edema", which would be understood by a clinician. However, the sentence does not contain names of these clinical conditions – rather it contains descriptions that imply the two conditions. Current literature on entity extraction has not paid much attention to implicit entities .

Another complexity in real-world scenarios and use cases is data heterogeneity due to their multimodal nature. There is an increasing availability of physical (including sensor/IoT), cyber, and social data that are related to events and experiences of human interest . For example, in our personalized digital health application for managing asthma in children[^5], we use numeric data from sensors for measuring a patient's physiology (e.g., exhaled nitric oxide) and immediate surroundings (e.g., volatile organic compounds, particulate matter, temperature, humidity), collect data from the Web for the local area (e.g., air quality, pollen, weather), and extract textual data from social media (i.e., tweets and web forum data relevant to asthma) . Each of these modalities provides complementary information that is helpful in evaluating a hypothesis provided by a clinician and also helps in disease management. We can also relate anomalies in the sensor readings (such as spirometer) to asthma symptoms and potential treatments (such as taking rescue medication). Thus, understanding a patient's health and well-being requires integrating and interpreting multimodal data and gleaning insights to provide reliable situational awareness and decisions. Knowledge bases play a critical role in establishing relationships between multiple data streams of diverse modalities, disease characteristics and treatments, and in transcending multiple abstraction levels . For instance, we can relate the asthma severity level of a patient, measured exhaled nitric oxide, relevant environmental triggers, and prescribed asthma medications to one another to come up with personalized actionable insights and decisions.

Knowledge bases can come in handy when there is not enough hand-labaled data for supervised learning. For example, emoji sense disambiguation, which is the ability to identify the meaning of an emoji in the context of a message in a computational manner , is a problem that can be solved using supervised and knowledge-based approaches. However, there is no hand-labeled emoji sense dataset in existence that can be used to solve this problem using supervised learning algorithms. One reason for this could be that emoji have only recently become popular, despite having been first introduced in the late 1990s . We have developed a comprehensive emoji sense knowledge base called EmojiNet  by automatically extracting emoji senses from open web resources and integrating them with BabelNet. Using EmojiNet as a sense inventory, we have demonstrated that the emoji sense disambiguation problem can be solved with carefully designed knowledge bases, obtaining promising results .

In this paper, we argue that careful exploitation of knowledge can greatly enhance the current ability of (big) data processing. At Kno.e.sis, we have dealt with several complex situations where:

1.  Large quantities of hand-labeled data required for unsupervised (self-taught) techniques to work well is not available or the annotation effort is significant.

2.  The text to be recognized is complex (i.e., beyond simple entity - person/location/organization), requiring novel techniques for dealing with complex/compound entities , implicit entities , and subjectivity (emotions, intention) .

3.  Multimodal data – numeric, textual and image, qualitative and quantitative, certain and uncertain – are available naturally .

Our recent efforts have centered around exploiting different kinds of knowledge bases and using semantic techniques to complement and enhance ML, statistical techniques, and NLP. Our ideas are inspired by the human brain's ability to learn and generalize knowledge from a small amount of data (i.e., humans do not need to examine tens of thousands of cat faces to recognize the next "unseen" cat shown to them), analyze situations by simultaneously and synergistically exploiting multimodal data streams, and understand more complex and nuanced aspects of content, especially by knowing (through common-sense knowledge) semantics/identity preserving transformations.

# Challenges in creating and using knowledge bases

Last decade saw an increasing use of background knowledge for solving diverse problems. While applications such as searching, browsing, and question answering can use large, publically available knowledge bases in their current form, others like movie recommendation, biomedical knowledge discovery, and clinical data interpretation are challenged by the limitations discussed below.
**Lack of organization of knowledge bases:** Proper organization of knowledge bases has not kept pace with their rapid growth, both in terms of variety and size. Users find it increasingly difficult to find relevant knowledge bases or relevant portions of a large knowledge base for use in domain-specific applications (e.g., movie, clinical, biomedical). This highlights the need to identify and select relevant knowledge bases such as the linked open data cloud, and extract the relevant portion of the knowledge from broad coverage sources such as Wikipedia and DBpedia. We are working on automatically indexing the domains of the knowledge bases  and exploiting the semantics of the entities and their relationships to select relevant portions of a knowledge base .
**Gaps in represented knowledge:** The existing knowledge bases can be incomplete with respect to a task at hand. For example, applications such as computer assisted coding (CAC) and clinical document improvement (CDI) require comprehensive knowledge about a particular domain (e.g., cardiology, oncology)[^6]. We observe that although the existing medical knowledge bases (e.g., Unified Medical Language System (UMLS)) are rich in taxonomical relationships, they lack non-taxonomical relationships among clinical entities. We have developed data-driven algorithms that use real-world clinical data (such as EMRs) to discover missing relationships between clinical entities in existing knowledge base, and then get these validated by a domain-expert-in-the-loop . Yet another challenge is creating personalized knowledge bases for specific tasks. For example, in , personal knowledge graphs are created based on the content consumed by a user, taking into account the dynamically changing vocabulary, and this is applied to improve subsequent filtering of relevant content.
**Inefficient metadata representation and reasoning techniques:** The scope of what is captured in the knowledge bases is rapidly expanding, and involves capturing more subtle aspects such as subjectivity (intention, emotions, sentiments), spatial and temporal information, and provenance. Traditional triple-based representation languages developed by Semantic Web community (e.g., RDF, OWL) are unsuitable for capturing such metadata due to their limited expressivity. For example, representation of spatio-temporal context or uncertainty associated with a triple is *ad hoc*, inefficient, and lacks semantic integration for formal reasoning. These limitations and requirements are well-recognized by the Semantic Web community, with some recent promising research to address them. For example, the singleton-property based representation  adds ability to make statements about a triple (i.e., to express the context of a triple) and probabilistic soft logic  adds ability to associate the probability value with a triple and reason over them. It will be really exciting to see applications exploiting such enhanced hybrid knowledge representation models that perform 'human-like' reasoning on them.

Next, we discuss several applications that utilize knowledge bases and multimodal data to circumvent or overcoming some of the aforementioned challenges due to insufficient manually-created knowledge.
**Application 1: Emoji sense disambiguation**

With the rise of social media, "emoji" have become extremely popular in online communication. People are using emoji as a new language on social media to add color and whimsiness to their messages. Without rigid semantics attached to them, emoji symbols take on different meanings based on the context of a message. This has resulted in ambiguity in emoji use (see Figure ). Only recently have there been efforts to mimic NLP techniques used for machine translation, word sense disambiguation and search into the realm of emoji . The ability to automatically process, derive meaning, and interpret text fused with emoji will be essential as society embraces emoji as a standard form of online communication. Having access to knowledge bases that are specifically designed to capture emoji meaning can play a vital role in representing, contextually disambiguating, and converting pictorial forms of emoji into text, thereby leveraging and generalizing NLP techniques for processing richer medium of communication.

As a step towards building machines that can understand emoji, we have developed EmojiNet , the first machine readable sense inventory for emoji. It links Unicode emoji representations to their English meanings extracted from the Web, enabling systems to link emoji with their context-specific meanings. EmojiNet is constructed by integrating multiple emoji resources with BabelNet, which is the most comprehensive multilingual sense inventory available to-date. For example, for the emoji 'face with tears of joy' , EmojiNet lists 14 different senses, ranging from happy to sad. An application designed to disambiguate emoji senses can use the senses provided by EmojiNet to automatically learn message contexts where a particular emoji sense could appear. Emoji sense disambiguation could improve the research on sentiment and emotion analysis. For example, consider the emoji , which can take the meanings *happy* and *sad* based on the context in which it has been used. Current sentiment analysis applications do not differentiate among these two meanings when they process . However, finding the meanings of by emoji sense disambiguation techniques  can improve sentiment prediction. Emoji similarity calculation is another task that could be benefited by knowledge bases and multi-modal data analysis. Similar to computing similarity between words, we can calculate the similarity between emoji characters. We have demonstrated how EmojiNet can be utilized to solve the problem of emoji similarity . Specifically, we have shown that emoji similarity measures based on the rich emoji meanings available in EmojiNet can outperform conventional emoji similarity measures based on distributional semantic models and also helps to improve applications such as sentiment analysis .
**Application 2: Implicit entity linking**

As discussed, one of the complexities in data is the ability to express facts, ideas, and opinions in an implicit manner. As humans, we seamlessly express and infer implicit information in our daily conversations. Consider the two tweets *"Aren't we gonna talk about how ridiculous the new space movie with Sandra Bullock is?"* and *"I'm striving to be +ve in what I say, so I'll refrain from making a comment abt the latest Michael Bay movie"*. The first tweet contains an implicit mention of movie 'Gravity' and the second tweet contains an element of sarcasm and negative sentiment towards the movie 'Transformers: Age of Extinction'. Both the sentiment and the movie are implicit in the tweet. While it is possible to express facts, ideas, and opinions in an implicit manner, for brevity, we will focus on how knowledge aids in automatic identification of implicitly mentioned entities in text.

We define implicit entities as "entities mentioned in text where neither its name nor its synonym/alias/abbreviation or co-reference is explicitly mentioned in the same text". Implicit entities are a common occurrence. For example, our studies found that 21% of movie mentions and 40% of book mentions are implicit in tweets, and about 35% and 40% of 'edema' and 'shortness of breath' mentions are implicit in clinical narratives. There are genuine reasons why people tend to use implicit mentions in daily conversations. Here are few reasons that we have observed:

1.  To express sentiment and sarcasm : See above examples.

2.  To provide descriptive information : For example, it is a common practice to describe the features of an entity rather than simply list down its name in clinical narratives. Consider the sentence 'small fluid adjacent to the gallbladder with gallstones which may represent inflammation.' This sentence contains implicit mention of the condition cholecystitis ('inflammation in gallbladder' is recognized as cholecystitis) with its possible cause. The extra information (i.e., possible cause) in description can be critical in understanding the patient's health status and treating the patient. While it is feasible to provide these extra information with the corresponding explicit entity names, it is observed that clinical professionals prefer this style.

3.  To emphasize the features of an entity : Sometimes we replace the name of the entity with its special characteristics in order to give importance to those characteristics. For example, the text snippet "Mason Evans 12 year long shoot won big in golden globe" has an implicit mention of the movie 'Boyhood.' There is a difference between this text snippet and its alternative form "Boyhood won big in golden globe." The speaker is interested in emphasizing the distinct feature of the movie, which would have been ignored if he had used the name of the movie as in the second phrase.

4.  To communicate shared understanding : We do not bother spelling out everything when we know that the other person has enough background knowledge to understand the message conveyed. A good example is the fact that clinical narratives rarely mention the relationships between entities explicitly (e.g., relationships between symptoms and disorders, relationships between medications and disorders), rather it is understood that the other professionals reading the document have the expertise to understand such implicit relationships in the document.

Whenever we communicate, we assume common understanding or shared-knowledge with the audience. A reader who does not know that Sandra Bullock starred in the movie 'Gravity' and that it is a space exploration movie would not be able to decode the reference to the movie 'Gravity' in the first example; a reader who does not know about Michael Bay's movie release would have no clue about the movie mentioned in the second tweet; a reader who does not know the characteristics of the clinical condition 'cholecystitis' would not be able to decode its mention in the clinical text snippet shown above; a reader who is not a medical expert would not be able to connect the diseases and symptoms mentioned in a clinical narrative. **These examples demonstrate the indispensable value of domain knowledge in text understanding**. Unfortunately, state-of-the-art named entity recognition applications do not capture implicit entities . Also, we have not seen big data-centric or other approaches that can glean implicit entities without the use of background knowledge (that is already available (e.g., in UMLS) or can be created (e.g., from tweets and Wikipedia)).

The task of recognizing implicit entities in text demands comprehensive and up-to-date world knowledge. Individuals resort to a diverse set of entity characteristics to make implicit references. For example, references to the movie 'Boyhood' can use phrases like *"Richard Linklater movie"*, *"Ellar Coltrane on his 12-year movie role"*, *"12-year long movie shoot"*, *"latest movie shot in my city Houston"*, and *"Mason Evan's childhood movie"*. Hence, it is important to have comprehensive knowledge about the entities to decode their implicit mentions. Another complexity is the temporal relevancy of the knowledge. The same phrase can be used to refer to different entities at different points in time. For instance, the phrase *"space movie"* referred to the movie 'Gravity' in Fall 2013, while the same phrase in Fall 2015 referred to the movie 'The Martian'. On the flip side, the most salient characteristics of a movie may change over time and so will the phrases used to refer to it. In November 2014 the movie 'Furious 7' was frequently referred to with the phrase *"Paul Walker's last movie"*. This was due to the actor's death around that time. However, after the movie release in April 2015, the same movie was often mentioned through the phrase *"fastest film to reach the \$1 billion"*.

We have developed knowledge-driven solutions that decode the implicit entity mentions in clinical narratives  and tweets . We exploit the publicly available knowledge bases (only the portions that matches with the domain of interest) in order to access the required domain knowledge to decode implicitly mentioned entities. Our solution models individual entities of interest by collecting knowledge about the entities from these publicly available knowledge bases, which consist of definitions of the entities, other associated concepts, and the temporal relevance of the associated concepts. Figure  shows a snippet from generated entity model. It shows the models generated for movies 'Gravity', 'Interstellar', and 'The Martian'. The colored (shaded) nodes (circles) represent factual knowledge related to these movies extracted from DBpedia knowledge base and the uncolored nodes represent the contextual knowledge (time-sensitive knowledge) related to entities extracted from daily communications in Twitter. The implicit entity linking algorithms are designed to carefully use the knowledge encoded in these models to identify implicit entities in the text.

**Application 3: Understanding and analyzing drug abuse related discussions on web forums**

The use of knowledge bases to improve keyword-based search has received much attention from commercial search engines lately. However, the use of knowledge bases alone cannot solve complex, domain-specific information needs. For example, answering a complex search query such as "How are drug users overdosing on semi synthetic opioid Buprenorphine?" may require a search engine to be aware of several facts, including that Buprenorphine is a drug, that users refer to Buprenorphine with synonyms such as 'bupe', 'bupey', 'suboxone', and 'subbies', and the prescribed daily dosage range for Buprenorphine. The search engine should also have access to ontological knowledge as well as other "intelligible constructs" that are not typically modeled in ontologies, such as equivalent references to the frequency of drug use, the interval of use, and the typical dosage, to answer such complex search needs. At Kno.e.sis, we have developed an information retrieval system that integrates ontology-driven query interpretation with synonym-based query expansion and domain-specific rules to facilitate analysis of online web forums for drug abuse-related information extraction. Our system is based on a context-free grammar (CFG) that defines the interpretation of the query language constructs used to search for the drug abuse-related information needs and a domain-specific knowledge base that can be used to understand information in drug-related web forum posts. Our tool utilizes lexical, lexico-ontological, ontological, and rule-based knowledge to understand the information needs behind complex search queries and uses that information to expand the queries for significantly higher recall and precision (see Figure ) . This research  resulted in an unexpected finding of abuse of over the counter drug, which led to a FDA warning[^7].
**Application 4: Understanding city traffic using sensor and textual observations**

With increased urbanization, understanding and controlling city traffic flow has become an important problem. Currently, there are over 1 billion cars on the road network, and there has been a 236% increase in vehicular traffic from 1981 to 2001 . Given that road traffic is predicted to double by 2020, achieving zero traffic fatalities and reducing traffic delays are becoming pressing challenges, requiring deeper understanding of traffic events, and their consequences and interaction with traffic flow. Sensors deployed on road networks continuously relay important information about travel speed through certain road networks while citizen sensors (i.e., humans) share real-time information about traffic/road conditions on public social media streams such as Twitter. As humans, we know how to integrate information from these multimodal data sources: qualitative traffic event information to account for quantitative measured traffic flow (e.g., an accident reported in tweets can explain a slow-moving traffic nearby). However, current research on understanding city traffic dynamics either focuses only on sensory data or only on social media data but not both. Further, we use historical data to understand traffic patterns and exploit the complementary and corroborative nature of these multimodal data sources to provide comprehensive information about traffic.

One research direction is to create and materialize statistical domain knowledge about traffic into a machine-readable format. In other words, we want to define and establish associations between different variables (concepts) in the traffic domain (e.g., association between 'bad weather' and a 'traffic jam'). However, mining such correlations from data alone is neither complete nor reliable. We have developed statistical techniques based on probabilistic graphical models (PGMs)  to learn the structure (variable dependencies), leverage declarative domain knowledge to enrich and/or correct the gleaned structure due to limitations of a data-driven approach, and finally learn parameters for the updated structural model. Specifically, we use the sensor data collected by 511.org to develop an initial PGM that explains the conditional dependencies between variables in the traffic domain. Then we use declarative knowledge in ConceptNet to add/modify variables (nodes) and the type and the nature of conditional dependencies (directed edges) before learning parameters, thereby obtaining the complete PGM. Figure (a)(i) shows a snippet of ConceptNet and Figure (a)(ii) demonstrates the enrichment step of the developed model using the domain knowledge in ConceptNet .

Another research direction is to characterize a normal traffic pattern derived from sensor observations and then detect and explain any anomalies using social media data. We used a Restricted Switching Linear Dynamical System (RSLDS) to model normal speed and travel time dynamics and detect anomalies. Using speed and travel time data from each link, plus our common sense knowledge about the nature of expected traffic variations, we learn the parameters of the RSLDS model for each link. We then use a box-plot of the log likelihood scores of the various average speed traces with respect to the RSLDS model to learn and characterize anomalies for each link in the San Francisco Bay Area traffic data . Later, given a new traffic speed trace over a link, we can obtain its log likelihood score with respect to the RSLDS model for the particular day of the week and the hour of the day, to determine whether it is normal or anomalous. This anomalous traffic speed information is further correlated with traffic events extracted from Twitter data (using crawlers seeded with OSM, 511.org and Scribe vocabularies) using their spatio-temporal context to explain the anomalies. Figure (b) demonstrates this process. **This example again demonstrates the vital role of multi-modal data for better interpretation of traffic dynamics, synthesizing probabilistic/statistical knowledge, and the application of both statistical models such as RSLDS and complementary semantic analysis of Twitter data**. Further exploration of different approaches to represent and exploit semantics appear in . Table  summarizes the role of knowledge bases in the four applications discussed above.

| **Problem Domain** | **Use of Knowledge bases** | **Nature of Improvement** |
|:---|:---|:---|
| Emoji Similarity and | Generation and application of | Leveraging linguistic knowledge |
| Sense Disambiguation | EmojiNet | for emoji interpretation |
| Implicit Entity Linking | Adapted UMLS definitions for | Recall and coverage |
|  | identifying medical entities, and |  |
|  | Wikipedia and Twitter data for |  |
|  | identifying Twitter entities |  |
| Understanding Drug | Application of Drug Abuse | Recall and coverage |
| Abuse-related | Ontology along with slang term |  |
| Discussions | dictionaries and grammar |  |
| Traffic Data Analysis | Statistical knowledge extraction | Anomaly detection and |
|  | and using ontologies for Twitter | explanation; Multi-modal data |
|  | event extraction | stream correlation |

Summary of knowledge-based approaches and the resulting improvements for each problem domain.

# Looking forward

We discussed the importance of domain/world knowledge in understanding complex data in the real world, particularly when large amounts of training data are not readily available or it is expensive to generate. We demonstrated several applications where knowledge plays an indispensable role in understanding complex language constructs and multimodal data. Specifically, we have demonstrated how knowledge can be created to incorporate a new medium of communication (such as emoji), curated knowledge can be adapted to process implicit references (such as in implicit entity and relation linking), statistical knowledge can be synthesized in terms of normalcy and anomaly and integrated with textual information (such as in traffic context), and linguistic knowledge can be used for more expressive querying of informal text with improved recall (such as in drug related posts). We are also seeing early efforts in making knowledge bases dynamic and evolve to account for the changes in the real world[^8].

Knowledge seems to play a central role in human learning and intelligence, such as in learning from a small amount of data, and in cognition – especially perception. Our ability to create or deploy just the right knowledge in our computing processes will improve machine intelligence, perhaps in a similar way as knowledge has played a central role in human intelligence. As a corollary to this, two specific advances we expect are: a deeper and nuanced understanding of content (including but not limited to text) and our ability to process and learn from multimodal data at a semantic level (given that concepts manifest very differently at the data level in different media or modalities). The human brain is extremely adept at processing multimodal data – our senses are capable of receiving 11 million bits per second, and our brain is able to distill that into abstractions that need only a few tens of bits to represent (for further explorations, see ). Knowledge plays a central role in this abstraction and reasoning process known as *the perception cycle*.

Knowledge-driven processing can be viewed from three increasingly sophisticated computational approaches: (1) Semantic Computing, (2) Cognitive Computing, and (3) Perceptual Computing. *Semantic Computing* refers to computing the type of a data value, and relating it to other domain concepts. In the healthcare context, this can involve relating symptoms to diseases and treatments. Ontologies, and Semantic Web technologies provide the foundation for semantic computing. *Cognitive computing* refers to representation and reasoning with data using background knowledge reflecting how humans interpret and process data. In the healthcare context, this requires capturing the experience and domain expertise of doctors through knowledge bases and heuristic rules for abstracting multimodal data into medically relevant abstractions, insights, and actions, taking into account triggers, personal data, patient health history, demographics data, health objectives, and medical domain knowledge. For instance, "normal'' blood pressure varies with factors such as age, gender, emotional state, activity, and illness; similarly, the "target" blood pressure, HBA1C, and cholesterol values a patient is advised to maintain depend on whether the patient is diabetic or not. In the traffic context, this can be used to interpret and label a time-series of traffic sensor data using a traffic event ontology. *Perceptual computing*, which builds on background knowledge created for semantic and cognitive computing, uses deductive reasoning to predict effects and treatments from causes, and abductive reasoning to explain the effects using causes, resolving any data incompleteness or ambiguity by seeking additional data. The knowledge itself can be a hybrid of deterministic and probabilistic rules, modeling both normalcy and anomalies, transcending abstraction levels. This directly contributes to making decisions and taking actions.

We expect more progress in hybrid knowledge representation and reasoning techniques to better fit domain characteristics and applications. Even though deep learning techniques have made incredible progress in machine learning and prediction tasks, they are still uninterpretable and prone to devious attacks. There are anecdotal examples of misinterpretations of audio and video data through adversarial attacks that can result in egregious errors with serious negative consequences. In such scenarios, we expect hybrid knowledge bases to provide a complementary foundation for reliable reasoning. In the medical domain, the use of interleaved abductive and deductive reasoning (a.k.a., perception cycle) can provide actionable insights ranging from determining confirmatory laboratory tests and disease diagnosis to treatment decisions. Declarative medical knowledge bases can be used to verify the consistency of an EMR and data-driven techniques can be applied to a collection of EMRs to determine and fix potential gaps in the knowledge bases. Thus, there is a symbiotic relationship between the application of knowledge and data to improve the reliability of each other. The traffic scenario shows how to hybridize complementary statistical knowledge and declarative knowledge to obtain an enriched representation (See also ). It also shows how multimodal data streams can be integrated to provide more comprehensive situational awareness.

Machine intelligence has been the holy grail of a lot of AI research lately. The statistical pattern matching approach and learning from big data, typically of a single modality, has seen tremendous success. For those of us who have pursued brain-inspired computing approaches, we think the time has come for rapid progress using a model-building approach. The ability to build broad models (both in terms of coverage as well as variety – not only with entities and relationships but also representing emotions, intentions and subjectivity features, such as, linguistic, cultural, and other aspects of human interest and functions) will be critical. Further, domain-specific, purpose-specific, personalized declarative knowledge combined with richer representation – especially probabilistic graph models – will see rapid progress. These will complement neural network approaches. We may also see knowledge playing a significant role in enhancing deep learning. Rather than the dominance of data-centric approaches, we will see an interleaving and interplay of the data and knowledge tracks, each with its own strengths and weaknesses, and their combinations performing better than the parts in isolation.

# Acknowledgments

We acknowledge partial support from the National Institutes of Health (NIH) award: 1R01HD087132-01: "kHealth: Semantic Multisensory Mobile Approach to Personalized Asthma Care" and the National Science Foundation (NSF) award: EAR 1520870: "Hazards SEES: Social and Physical Sensing Enabled Decision Support for Disaster Management and Response". Points of view or opinions in this document are those of the authors and do not necessarily represent the official position or policies of the NIH or NSF.

[^1]: <http://j.mp/15yrsSS>

[^2]: <https://en.wikipedia.org/wiki/Siri>

[^3]: <http://bit.ly/22xUjZ6>

[^4]: <https://goo.gl/A54hno>

[^5]: <http://bit.ly/kAsthma>

[^6]: <https://goo.gl/nXDY8x>

[^7]: <http://bit.ly/k-FDA>

[^8]: <http://bit.ly/2cVGbov>


abstract: It is known that Einstein's conceptual base for his theory of relativity was the philosophy formulated by Immanuel Kant. Things appear differently to observers in different frames. However, Kant's Ding-an-Sich leads to the existence of the absolute reference frame which is not acceptable in Einstein's theory. It is possible to avoid this conflict using the ancient Chinese philosophy of Taoism where two different views can co-exist in harmony. This is not enough to explain Einstein's discovery of the mass-energy relation. The energy-momentum relations for slow and ultra-fast particles take different forms. Einstein was able to synthesize these two formulas to create his energy-mass relation. Indeed, this is what Hegelianism is about in physics. Isaac Newton synthesized open orbits for comets and closed orbits for planets to create his second law of motion. Maxwell combined electricity and magnetism to create his four equations to the present-day wireless world. In order to synthesize wave and particle views of matter, Heisenberg formulated his uncertainty principle. Relativity and quantum mechanics are the two greatest theories formulated in the 20th Century. Efforts to synthesize these two theories are discussed in detail.

**Historical Approach to Physics according to Kant, Einstein, and Hegel**

Y. S. Kim
Center for Fundamental Physics, University of Maryland,
College Park, Maryland, 20742, U.S.A.

based on an invited presented at the 32nd Congress of the Italian Society of Historians of Physics and Astronomy (Rome, Italy, September 2012).

# Introduction

Einstein studied the philosophy of Immanuel Kant during his earlier years. It is thus not difficult to see he was influenced by Kantian view of the world when he formulated his special theory of relativity. It is also known that, in formulating his philosophy, Kant was heavily influenced by the environment of Koenigsberg where he spent eighty years of entire life. The first question is what aspect of Kant's city was influential to Kant. We shall start with this issue in this report.

In Einstein's theory, one object looks differently to moving observers with different speeds. This aspect is quite consistent with Kant's philosophy. According to him, one given object or event can appear differently to observers in different environments or with different mindsets. In order to resolve this issue, Kant had to introduce the concept of "Ding-an-Sich" or thing in itself meaning an ultimate object of absolute truth. Indeed, Kant had a concept of relativity as Einstein did, but his Ding-an-Sich led to the absolute frame of reference. Here Kantianism breaks down in Einstein's theory. Kant's absolute frame does not exist according to Einstein. In order to resolve this issue, let us go to the ancient Chinese philosophy of Taoism. Here, there are two different observers with two opposite points of view. However, this world works when these two views form a harmony. Indeed, Einsteinism is more consistent with Taoism. The energy-momentum relations is different for a massive -slow particle and for a fast-massless particle. Einstein's relativity achieved the harmony between these two formulas.

This leads us to Hegel's approach to the world. If there are two opposite things, it is possible to derive a new thing from them. This is what Einsteinism is all about. Einstein derived his $E = mc^2$ from two different expressions of the energy-momentum relation for massive and massless particles.

Einstein thus started with Kantianism, but he developed a Hegelian approach to physical problems. Indeed, this encourages us to see how this Hegelianism played the role in developing new laws of physics. For instance, Newton's equation of motion combines the open orbits for comets and the closed orbits of planets.

If this Hegelian approach is so natural to the history of physics, there is a good reason. Hegel derived his philosophy by studying history. Hegel observed that Christianity is a product of Jewish one-God religion and Greek philosophy. Since Hegel did not understand physics, his reasoning was based on historical development of human relations. It is thus interesting proposition to interpret Hegel's philosophy using the precise science of Physics.

In Secs. 2 and 3, we review how Kantianism was developed and how Einstein was influenced by Kant. In Sec. 4, it is pointed out that Hegelianism is the natural language in understanding physics. In Sec. 5, we examine whether quantum mechanics and relativity can be combined into one theory according to Hegelian approach to the history of physics.

# Geographic Origin of Kantianism and Taoism

Immanuel Kant (1724-1804) was born in the East Prussian city of Koenigsberg, and there he spent 80 years of his entire life. It is agreed that his philosophy was influenced by the lifestyle of Koenigsberg.

The city of Koenigbeerg is located at the Baltic wedge between Poland and Lithuania. As shown in Fig. 1, this place served as the traffic enter for maritime traders in the Baltic Sea. In addition, this city is between the eastern and western worlds (Applebaum 1994). However, there are no natural boundaries such a rivers or mountains. Thus, anyone with a stronger army could come to this area and run the place.

Indeed, Koenigsberg was a meeting place for many people with different ideas and different view points. Kant observed that the same thing can appear differently depending on the observer's location or state of mind.

The basic ingredients of Taoism are known to be two opposite elements Yang (plus) and Ying (minus). This world works best if these two elements form a harmony. However, the most interesting aspect of Taoism is that its geographic origin is the same as that of Kantianism. Let us look at the map of China given in Fig 1during the period from 1600 to 100 BC. China started as collection of isolated pockets of population. They then came to banks of the Yellow River, and started to communicate with those from other areas. They drew pictures for written communication leading eventually to Chinese characters.

How about different ideas? They grouped many different opinions into two groups, leading to the concept of Yang and Ying. Immanuel Kant considered many different views, but he concluded that there must be one and only truth.

Indeed, Taoism and Kantianism started with the same environment, but Kant insisted on one truth called Ding-an-Sich, while Taoism ended up with two opposing elements (Kim 2006).

# Kantian Influence on Einstein

During his early years, Einstein became quite interested in Kant and studied his philosophy rigorously. This was quite common among the young students during his time. Einstein however studied also physics, and got an idea that one object could appear differently for observers moving with different speeds.

Let us go to Fig. 2. According Kant, an object or event look differently to different observers depending on their places or states of mind. A Coca-Cola can looks like a circle if viewed from the top. It appears like a rectangle if viewed from the side. The Coca-Cola can is an absolute thing or his Ding-an-Sich. Likewise, the electron orbit of the hydrogen atom looks like a circle for an observer when both the hydrogen and the observer are stationary. If the hydrogen atom in on a train, our first guess is that it should look like an ellipse. This is what Einstein inherited from Kant.

However, does the hydrogen atom require a Ding-an-Sich? The answer is No. Indeed, Kant attempted to formulate his theory of relativity with an absolute coordinate system corresponding to his Ding-an-Sich. This is the basic departure of Einsteinism from Kantianism

Let us come back to Einstein. Like Kant, Einstein started from different observers looking at a thing differently, but ended up with a particle at the rest and the same particle moving with a speed close to that of light. He then derived his celebrated energy-mass relation, as indicated in Fig. 4 (left). Einstein had to invent a formula applicable to both. This is precisely a Hegelian approach to physics. It is not clear whether Einstein knew he was doing Hegel. This remains as an interesting historical problem. As for Einstein's hydrogen atom, we now have hadrons which are bound states of quarks, while the hydrogen atom is a bound state of a proton and an electron. The proton is a hadron and is a charged particle which can be accelerated to the speed very close to that of light. We shall return in Sec. 5 to the problem presented in the right side of Fig. 3.

However, does the hydrogen atom require a Ding-an-Sich? The answer is No. Indeed, Kant attempted to formulate his theory of relativity with an absolute coordinate system corresponding to his Ding-an-Sich. This is the basic departure of Einsteinism from Kantianism Let us come back to Einstein. Like Kant, Einstein started from different observers looking at a thing differently, but ended up with a particle at the rest and the same particle moving with a speed close to that of light. He then derived his celebrated energy-mass relation, as indicated in Fig. 4 (left).

Einstein had to invent a formula applicable to both. This is precisely a Hegelian approach to physics. It is not clear whether Einstein knew he was doing Hegel. This remains as an interesting historical problem. As for Einstein's hydrogen atom, we now have hadrons which are bound states of quarks, while the hydrogen atom is a bound state of a proton and an electron. The proton is a hadron and is a charged particle which can be accelerated to the speed very close to that of light. We shall return in Sec. 5 to the problem presented in the right side of Fig. 3.

# Hegelian Approach to the History of Physics

Since Hegel formulated his philosophy while studying history, it is quite natural to write the history of physics according to Hegel. First of all, Isaac Newton combined hyperbolic-like orbits for comets and elliptic orbits for planets to derive his second-order differential equation which is known today as the equation of motion.

James Clerk Maxwell combined the theory of electricity and that for magnetism to formulate his electromagnetic theory leading to the present world of wireless communication. Max Planck observed that the radiation laws are different for low- and high-frequency limits. By deriving one formula for both, he discovered Planck's constant.

Werner Heisenberg observed the matter appears as a particle also appears as a wave, with entirely different properties. He found the common ground for both. In so doing, he found the uncertainty relation which constitutes the foundation of quantum m echanics.

Indeed, quantum mechanics and relativities were two most fundamental theories formulated in the twentieth century. They were developed independently. The question is whether they can be combined into one theory. We shall examine how Hegelian approach is appropriate for this problem in Sec. 5.

# How to combine Quantum Mechanics and Relativity

Here again, the problem gets divided into scattering and bound state problems. Quantum field theory was developed for scattering problems, and this theory is accepted as a valid theory, as is illustrated in Fog. 4.

For bound-state problems, Paul A. M. Dirac wrote three important papers on this subject (Dirac 1927,1945, 1949). His 1927 paper tells us there is a time-energy uncertainty relation. In 1945, he attempted to use the harmonic oscillator to formulate quantum mechanics applicable to Einstein's world. If we combine or Hegelize Dirac's 1927 and 1945 papers, we end up with the circle given in Fig. 5.

In 1949, Dirac showed the Lorentz boost can be described as a squeeze transformation as shown in Fig. 5. If we Hegelize the circle and the squeezed rectangle, we arrive at the ellipse (Kim and Noz 2011) which can explain what happens in the real world including the quark model (Gell-Mann 1964) and the parton model (Feynman 1969). This Hegelian procedure corresponds to Step 1 in Fig. 4.

The final step in constructing Lorentz-covariant quantum mechanics is to show that the scattering and bound states share the same set of fundamental principles (Han et al. 1981). This Hegelians procedure is illustrated in as Step 2 in Fig. 4.

# Kant, Hegel, and Einstein

Kant and Hegel are two of the most fundamental thinkers affecting our present-day lifestyles. However, their philosophies were based largely on social events and applicable to formulation of social sciences. It is gratifying to note that Einstein gives us a more concrete picture of their approaches to the problems. By building the bridge between Kant and Hegel as illustrated in Fig. 6, Einstein not only gives us the precise description of how physical theories were developed in the past but also tells us how to approach current problems in physics.