Reviewerly/exHarmony huggingface dataset:
Memory-based Consultation System for Personalized Conversations Using Temporal Contexts

Due to the rapid development of natural language understanding technology, there are many attempts to apply interactive conversational systems in various domains such as consultation. In a conversational system, the quality of the generated dialogue has an important influence on the user's satisfaction, and its ability to lead a specialized conversation is required for the specific user. In this paper, we design a conversational system to manage user-specific memory information extracted from a conversation history corresponding to the user in the field of consultation. At the same time, the proposed system is configured to grasp the contextual information in the conversation history based on a language model learned to extract temporal contexts. We utilize the contexts and keywords to transform the major contents of the conversation history into episodic memory. By accumulating the episodic memories for the particular user and structuring them as long-term memory, we develop the basis to provide personalized conversation to the user. Based on the memory built for each user, we believe that customization can be possible depending on the user's historical information in the process of dialogue generation.
Improving Zero-Shot Inference with Unsupervised Key-Sentences Extraction

In many text applications, low availability of labeled data is common, which makes training machine learning models unfeasible. Many approaches have been proposed to try to circumvent this limitation, with emphasis on zero-shot models. Zero-shot models can perform inference without training. However, they tend to have lower performance in long texts. In this work, we propose to apply an unsupervised technique to rank the most relevant sentences of the long text and apply zero-shot models to these most relevant sentences. Tests on the text classification task demonstrate a performance gain in different types of zero-shot models and different languages.
Visual Enhancement and Semantic Segmentation of Murine Tissue Scans with Pulsed THz Spectroscopy

Semantic Artificial Intelligence has certain qualities that are advantageous for deep learning-based medical imaging tasks. Medical images can be augmented by injecting semantic context into the underlying classification mechanism, increasing the information density of the scan and ultimately can provide more trust in the result. This work considers an application of semantic AI to segment tissue types from excised breast tumors imaged with pulsed terahertz (THz)–an emerging imaging technology. Prior work has demonstrated traditional data driven methodology for deep learning on THz has two key weaknesses: namely 1) low image resolution compared to other state-of-the-art imaging techniques and 2) a lack of expertly-labeled images due to domain transformation and tissue changes during histopathology. This work seeks to address these limitations through two semantic AI mechanisms. Specifically, we introduce a two stage pipeline using an unsupervised image-to-image translation network and a supervised segmentation network. The combination of these contributions enables enhanced near-real-time visualization of excised tissue through THz scans and a supervised segmentation and classification training strategy using only synthetic THz scans generated by our bi-directional image-to-image translation network.
Ontology Modeling for Probabilistic Knowledge Graphs

Reasoning about data given uncertain future conditions is a crucial component of modern decision support systems. While ontologies allow domain-specific data to be represented in knowledge graphs, they generally represent data in an absolute way and do not support reasoning under uncertain conditions. In this paper, we present a novel technique for using a probabilistic ontology to extend knowledge graphs as Bayesian networks, which is a step towards enabling probabilistic reasoning under uncertainty within a knowledge graph. Although prior work has focused on using the semantic relationships in an ontology to automatically infer the structure of a Bayesian network, we additionally emphasize the ontological modeling of the Bayesian network as a layer in the knowledge graph itself. We define a custom probabilistic ontology that describes the requisite probabilistic elements, including Random Variables, the conditional dependencies between them, and their distributions. It also includes graph structures for representing decision optimization under uncertainty. Our technique is generalized to work regardless of the domain of the data. In the evaluation, we demonstrate that our approach can scale over large data volumes, and that integrating the probabilistic elements with the domain ontology allows for flexible queries that can answer competency questions more efficiently than queries over the domain ontology alone.
Actionable Suggestions in Support of Rehospitalization Risk Predicted by Artificial Intelligence

Discharging a patient from hospital is a crucial decision, as wrong timing may lead to rehospitalization. In this scenario, Artificial Intelligence (AI) can support physicians in the decision-making process. However, AI provides a probability that an event will happen, without information regarding how to reduce such probability. EXplainable Artificial Intelligence (XAI) methods, allow to better understand how the score is computed but still lack in providing specific suggestions. We propose a method that creates a set of actionable suggestions on how to change the feature values to reduce a risk predicted by AI, and support the physicians in deciding when to discharge a patient. We utilize historical data distribution of features to understand how values for the current case should be modified. After retrieving a set of relevant historical data by considering the similarity with the current point, we use statistical analysis to highlight differences between the feature values of current and historical cases. A suggestion is made whenever a significative difference on a controllable feature is found. For the evaluation, we used an electronic health record (MIMIC-III) to produce suggestions to reduce 30 days rehospitalization probability. A ground truth set of suggestions was created for 54 patients, by considering how changes in feature values affected the risk, as well as if feature values fell within literature ranges. The proposed approach reached a precision of 80%. The adoption of this method will allow to decrease costs related to rehospitalizations and positively impact patients’ health.
Can Eye Gaze Improve Emotional State Detection on Off the Shelf Smart Devices

Smartphones and wearable technology have revolutionized digital healthcare through the use of rich sensor data. Digital phenotyping, a field that uses smartphone data to detect or recognize cognitive, behavioral, or affective states and traits, is often based on data from sensors (such as inertial or touch sensors), activity or user logs, and user-generated content. In this paper, we propose the use of eye gaze as a new digital biomarker for affect detection, leveraging the advanced capabilities of off the-shelf smart devices. We designed two studies in the Instagram use scenario to detect emotional state change using gaze data. The first study took place in a controlled setting and help us understand the value of gaze features for affect detection. In this study, we achieved a peak accuracy of 76.4% for binary valence detection. The second study will be conducted over a longer period in real-world settings, with a larger population, to assess the effectiveness of our approach. By including this new sensing modality in affective digital phenotyping, we plan to improve the reliability and robustness of emotional state detection.
Retrieving Users’ Opinions on Social Media with Multimodal Aspect-Based Sentiment Analysis

People post their opinions and experiences on social media, yielding rich databases of end-users’ sentiments. This paper shows to what extent machine learning can analyze and structure these databases. An automated data analysis pipeline is deployed to provide insights into user-generated content for researchers in other domains. First, the domain expert can select an image and a term of interest. Then, the pipeline uses image retrieval to find all images showing similar content and applies aspect-based sentiment analysis to outline users’ opinions about the selected term. As part of an interdisciplinary project between architecture and computer science researchers, an empirical study of Hamburg’s Elbphilharmonie was conveyed. Therefore, we selected 300 thousand posts with the hashtag ‘hamburg’ from the platform Flickr. Image retrieval methods generated a subset of slightly more than 1.5 thousand images displaying the Elbphilharmonie. We found that these posts mainly convey a neutral or positive sentiment towards it. With this pipeline, we suggest a new semantic computing method that offers novel insights into end-users opinions, e.g., for architecture domain experts.
Supporting the construction of mystery novel knowledge graphs using BERT

In the Knowledge Graph Reasoning Challenge, the plot of a mystery novel is converted into a knowledge graph. Building the graph requires the work of choosing the parts to put into the knowledge graph. This research aims to automate this task. Specifically, BERT, a Natural Language Processing model, is used to classify with either summary or non-summary, with the results proposed as the parts to put into the knowledge graph. We established a binary classification model and verified its accuracy. Its was found to have an F-value of 0.59. Given the likelihood of improving the accuracy, it is thought that organizing the dataset could have a significant impact. This is to be pursued as a next step.
Lazy Node-Dropping Autoencoder

Autoencoders are widely used for dimensionality reduction nonlinearly. However, determining the number of nodes in the autoencoder embedding space is still a challenging task. The number of nodes in the bottleneck layer, which is an encoded representation, is estimated and determined by users. Therefore, to maintain embedding performance and reduce the complexity of the model, an indicator that automatically selects the number of bottleneck nodes is needed. This study proposes a method for automatically estimating the adequate number of nodes in the bottleneck layer while training the model. The basic idea of the proposed method is to eliminate lazy nodes which rarely affect the model performance based on the weight distribution of the bottleneck layer. Since the proposed method takes place in the learning process of the autoencoder, it has the advantage of accelerating the training speed. The proposed method showed better or similar performances in classification accuracy.
Semantic Validation and Interpretability of Object Detection

Object detection is an aspect of computer vision that can reach high levels of accuracy and is involved in many aspects of public interaction. Accuracy issues can arise when it comes to the application of these models to new data sets; explainability and interpretability of object detection models are still lacking which affects the trust non-experts have in the model’s decision-making process. In this paper, we define and test a method of semantic validation for object detection. With this system's implementation, the ability to accurately validate its detections jumps from an original 8% to 73% while still providing interpretability to its users.
Validating Security Requirement Specifications through the use of a Knowledge Graph

Turning a customer's product vision into implementable requirement specifications is notoriously difficult and time-consuming. Beyond that, security is an aspect of requirement specifications that is frequently underdeveloped. If developers had access to an automated system that could validate requirement specifications based on known loopholes and external expertise, the likelihood of injecting a security vulnerability into a software product would decrease. As depicted in this paper, the automated system takes a Software Requirement Specification (SRS) document as input. The system then potentially validates that all related security topics and vulnerabilities are covered within the security requirement specifications. Entity linking is used to parse the SRS to form queries for a security concepts and vulnerabilities knowledge graph. The query result can provide developers with areas of security to be considered as updates to the SRS and to help during the software design stage to improve the system's security.
OntoSeg - Segmentation of Large Volumetric Datasets Using Semantic Knowledge

The segmentation of a volumetric dataset, as they are commonly generated in computed tomography, into disjoint semantically understandable parts poses an ongoing problem. This is especially the case in the industrial domain of image processing for computed tomography, where the main problems lie in the wide range of applications together with the size of the generated datasets (up to Terabytes for a single scan). Both problems severely prohibit the processing of such data with established purely data-driven methods. This work proposes the OntoSeg system, which consists of a combination of semantic knowledge (in form of an ontology) with image processing methods. The latter are implemented in individual libraries, which may be attributed to any semantic component contained in the dataset. Thus, each library can be exchanged or enhanced flexibly and can incorporate arbitrary additional knowledge, which is hardly ever possible in classical monolithical image analysis systems. The proposed system incrementally restricts the dataset to the relevant regions, which allows for the segmentation of the final components with existing methods. An evaluation on a small ontology specific to the structure of cars enabled the efficient localization and segmentation of certain components, without having to segment every single of its billions of voxels.
AnyFace: A Data-Centric Approach For Input-Agnostic Face Detection

Face detection is a mandatory step in many computer vision applications, such as face recognition, emotion recognition, age detection, virtual makeup, and vital sign monitoring. Thanks to advancements in deep learning and the introduction of annotated large-scale datasets, numerous applications have been developed for human faces. Recently, other domains, such as animals and cartoon characters, have started gaining attention but still lag far behind human faces. The biggest challenge is the limited number of annotated face datasets in these domains. The manual labeling of large-scale datasets is tedious and requires substantial human labor. In this regard, we present an inputagnostic face detector to ease the annotation of various face datasets. We propose a simple but effective data-centric approach instead of building a specific neural network architecture. Specifically, we trained a face detection model, YOLO5Face, on human, animal, and cartoon face datasets. The experiments show that the model can achieve accurate results in all domains. In addition, the model achieved decent results for animals and cartoon characters different from the ones in the training set. This implies that the model can extract agnostic facial features. We have made the source code and pre-trained models publicly available at https://github.com/IS2AI/AnyFace to stimulate research in these fields.
Can CLIP Share Image in Dialogue?

Recently, many studies have constructed multimodal dialogue datasets containing image-sharing behavior, which is vital to increase the social relationship with interlocutors in open-domain conversation. In this paper, we report the empirical results that CLIP can understand the alignment between the dialogue history and image by conducting various experiments for (1) zero-shot transferability, (2) the effect of dialogue history, and (3) robustness. Our experiments demonstrate that it is necessary for improving the zero-shot performance of CLIP on the multi-modal dialogue dataset. Additionally, the CLIP model is benefitted from more informative texts (i.e., dialogue history), not the last utterance only.
Dynamic User Understanding and Interaction in Cultural Heritage Domain

We describe a module of a larger multimodal interactive environment that exploits a cognitive framework named VESSEL (Valuable Environment for Social Supporting and Empathetic Linking) in the cultural heritage domain. The module makes use of a conversational agent based on the RASA framework and a Compact Prediction Tree to learn points that can catch the user's attention during the visit according to her or his age and educational level.
Automatic Rule Definition for Pattern-Based Text Mining

Due to the exponential growth of the number of texts available online, automatic text mining techniques have been receiving much attention recently. Pattern-based text mining techniques have been specially studied because they allow users to extract points of interest from text automatically. However, the quality of obtained results is highly dependent on the quality of the rules that are used in the mining process. Currently, the rules that are more effective are the ones that are defined manually, but the creation of such rules is resource-consuming as it requires experts to define them. In this paper, we develop a method of automatic rule definition for pattern-based text mining. Our method is investigated in terms of aspect classification and sentiment analysis. We also examine combining pattern-based text mining with machine learning techniques. The evaluations were performed using two text data, namely, those mentioning API from Stack Overflow and those of Amazon review on PC peripherals. Evaluation results showed that meaningful rules were defined automatically by our method. We also show that our methodology can be used together with manually defined rules, resulting in outperforming other methods.
Prediction of Recurrence Probability of Thyroid Cancer Patients using Similarity Loss based Multi-modal Autoencoder

Data used for the diagnosis of thyroid cancer patients include Thyroid Stimulating hormone (TSH), Thyroglobulin (Tg), Tg Antibodies (TgAb), and pathology information. However, existing thyroid cancer-related studies using machine learning tended to use a single type of data in model. Therefore, this study proposes a new model and method that can use all type of data together. The proposed method uses an autoencoder model composed of multiple encoders to extract comprehensive features from three types of hormone related data and pathology data and connects them to a classification layer to train them to be classified. As a result of the experiment, the proposed model using both types of hormonal and pathological data showed a performance improvement of up to about 58.1 times compared to the model using only a single type of data.
Towards reasoning over knowledge graphs under aleatoric and epistemic uncertainty

The objective of this short article is to propose a research orientation to address the international Knowledge Graph Reasoning Challenge (IKGRC). The IKGRC consists in solving mysterious crimes and providing a reasonable explanation based on open knowledge graphs, which represent the case, background, and characters of some Sherlock Holmes novels.The suggestion of this paper is to use the framework of the belief functions, well adapted to the consideration of different uncertainties and the management of conflicting information.
Dialogue Response Evaluation Model with Conversational Feature Sensitive Negative Sampling

Evaluating the conversational responses is a challenging task. This is because there are so many possible responses in open-domain conversation. Recent work finds that appropriate negative samples are effective in practice, but there was problem that labeled conversational negative sample was insufficient. It is important to create an appropriate negative sample automatically because it is too costly to create all possible responses by person and annotate the response’s coherence score. To address this problem, we propose a method for generating and labeling feature sensitive negative responses for conversation automatically. Besides, we show that the model learned with the generated negative sample performs well with high correlation between human score and model score.
Criminal Investigation with Augmented Ontology and Link Prediction

This paper shows that our basic approach to the Knowledge Graph Reasoning Challenge is to create an ontology of motives and means. We also use link prediction based on knowledge graph embedding techniques to solve the case. The ontologies on motives and means are based on hand-crafted in our previous work and are supplemented by an automated ontology created by linking Wikidata. As a result of preliminary experiments, we present an overview of the ontology created at this stage and show how to predict characters with motives for incidents from the knowledge graph of the novel The Adventure of the Speckled Band.
Federated Learning Framework for Blockchain based on Second-Order Precision

FL has been gaining attention due to its ability to protect data privacy. By exchanging encrypted information between data providers, FL allows collaboration on global models without sharing local data. As many participants are involved, a centralized framework is not suitable. To decentralize FL, we propose BDFL, a novel architecture integrating FL into the blockchain consensus process. It uses a second-order algorithm to improve model convergence and reduce communication consumption, with the added benefit of handling data heterogeneity in Vertical FL. Blockchain is used to replace centralized parties, aggregating information and enabling traceability of malicious parties. Experimental results show that BDFL is faster and more communication-efficient than traditional algorithms, and robust to malicious nodes.
Attribute Enhancement using Aligned Entities between Knowledge Graphs

A knowledge graph (KG) is a structural form of semantic network that integrates triplets into a graph to support knowledge processing and reasoning. Popular KGs like DBpedia, YAGO, and Freebase have mostly been constructed by using automatic extraction methods on crowd-sourced content; therefore, they are not always complete or error-free. In this study, we propose an attribute enhancement framework (AEF) to enrich the attributes of entities by integrating multiple KGs based on their aligned information. Usually, similar entities from different KG contain different sets of attributes. AEF uses a representation-learning-based ranking model to determine the significant attributes. It employs a similarity mapping method to integrate new attributes into the target KG. AEF can also identify attribute value inconsistencies between two KGs. In our experimental dataset, 46% new attribute properties and 29% new values from DBpedia are proposed for the YAGO entities. In this study, we aim to include all the important attributes in the existing KGs with the goal of making a more robust and complete knowledge graph.
A Method to Constract a Masked Knowlege Graph Model using Transformer for Knowledge Graph Reasoning

Most of the previous methods using machine learning for this challenge generate a new knowledge graph from the original one, and some information is lost in the process of creating a new knowledge graph. Therefore, we proposed a new model to estimate the criminal without changing the original knowledge graph. The proposed model uses a Transformer and allows the estimation of unknown criminals in nonexistent scenes by learning similar to Masked Language Modeling in BERT. This model, which uses the original knowledge graph, is expected to infer information about the crime scene at the same time as predicting the criminal. We confirmed by experiments that the model had gained the ability to estimate the hidden story parts by considering the surrounding stories.