Reviewerly/exHarmony huggingface dataset:
Optimal transport for conditional domain matching and label shift

We address the problem of unsupervised domain adaptation under the setting of generalized target shift (joint class-conditional and label shifts). For this framework, we theoretically show that, for good generalization, it is necessary to learn a latent representation in which both marginals and class-conditional distributions are aligned across domains. For this sake, we propose a learning problem that minimizes importance weighted loss in the source domain and a Wasserstein distance between weighted marginals. For a proper weighting, we provide an estimator of target label proportion by blending mixture estimation and optimal matching by optimal transport. This estimation comes with theoretical guarantees of correctness under mild assumptions. Our experimental results show that our method performs better on average than competitors across a range domain adaptation problems including digits,VisDA and Office. Code for this paper is available at https://github.com/arakotom/mars_domain_adaptation .
Reliable shot identification for complex event detection via visual-semantic embedding

Multimedia event detection is the task of detecting a specific event of interest in an user-generated video on websites. The most fundamental challenge facing this task lies in the enormously varying quality of the video as well as the high-level semantic abstraction of event inherently. In this paper, we decompose the video into several segments and intuitively model the task of complex event detection as a multiple instance learning problem by representing each video as a "bag" of segments in which each segment is referred to as an instance. Instead of treating the instances equally, we associate each instance with a reliability variable to indicate its importance and then select reliable instances for training. To measure the reliability of the varying instances precisely, we propose a visual-semantic guided loss by exploiting low-level feature from visual information together with instance-event similarity based high-level semantic feature. Motivated by curriculum learning, we introduce a negative elastic-net regularization term to start training the classifier with instances of high reliability and gradually taking the instances with relatively low reliability into consideration. An alternative optimization algorithm is developed to solve the proposed challenging non-convex non-smooth problem. Experimental results on standard datasets, i.e., TRECVID MEDTest 2013 and TRECVID MEDTest 2014, demonstrate the effectiveness and superiority of the proposed method to the baseline algorithms.
Altering Non-verbal Cues to Implicitly Direct Attention in Social VR

In this work we explore a concept system that alters the virtual eye movements without the user’s awareness, and whether this can affect social attention among others. Our concept augments the real movements with subtle redirected gazes to people, that occur in intervals to remain unnoticed. We present a user study with groups of people conversing on a topic, and measure the level of visual attention among users. Compared to a baseline of natural eye movements, we find that the method has indeed affected the overall attention in the group, but in unexpected ways. Our work points to a new way to exploit the inherent role of eyes in social virtual reality.
Incorporating Query Reformulating Behavior into Web Search Evaluation

While batch evaluation plays a central part in Information Retrieval (IR) research, most evaluation metrics are based on user models which mainly focus on browsing and clicking behaviors. As users' perceived satisfaction may also be impacted by their search intent, constructing different user models across various search intent may help design better evaluation metrics. However, user intents are usually unobservable in practice. As query reformulating behaviors may reflect their search intents to a certain extent and highly correlate with users' perceived satisfaction for a specific query, these observable factors may be beneficial for the design of evaluation metrics. How to incorporate the search intent behind query reformulation into user behavior and satisfaction models remains under-investigated. To investigate the relationships among query reformulations, search intent, and user satisfaction, we explore a publicly available web search dataset and find that query reformulations can be a good proxy for inferring user intent, and therefore, reformulating actions may be beneficial for designing better web search effectiveness metrics. A group of Reformulation-Aware Metrics (RAMs) is then proposed to improve existing click model-based metrics. Experimental results on two public session datasets have shown that RAMs have significantly higher correlations with user satisfaction than existing evaluation metrics. In the robustness test, we have found that RAMs can achieve good performance when only a small proportion of satisfaction training labels are available. We further show that RAMs can be directly applied in a new dataset for offline evaluation once trained. This work shows the possibility of designing better evaluation metrics by incorporating fine-grained search context factors.
Coordinating Experience Replay: A Harmonious Experience Retention approach for Continual Learning

Continual Learning (CL) aims at learning new knowledge gradually from an infinite data streams while retaining the old knowledge. A key problem in CL is the stability–plasticity dilemma, i.e., a high stability weakens the model from learning new data, while a high plasticity results in forgetting old classes. To alleviate this dilemma, some work has been reported to replay a finite subset of stored examples when training on new tasks, while the effectiveness heavily depends on the rehearsal mode and the validity of these exemplars. To this end, this paper proposes a Coordinating Experience Replay approach consisting of a Harmonious Experience Retention approach and a Vital Exemplar Reserved Sampling strategy, which constrains the rehearsal process and develops two criteria to select examples for better replaying the beneficial experience. Specifically, for the Harmonious Experience Retention approach, an indirect rehearsal process is supplemented on Dark Experience Replay by Imitative Experience Replay (IER), which imitates the response process via constructing empirical constraints on the saved logits and ground-truth labels directly. For the Vital Exemplar Reserved Sampling strategy, we apply the classification loss value and the class information as the criteria to select examples, which balance the criteria of class integrity and example vitality. Extensive experiments on seven benchmark datasets show the superiority of the proposed approach under diverse continual learning settings.
Adaptive test case selection for DNN-based perception functions

The progress in deep learning methods has bolstered the development of automated vehicles during the last decade. However, the deployment of deep learning methods in safety-critical applications raised questions on their safety. Like other vehicle components, a testing process has to prove the reliability of perception systems. Scalability issues arise when using real-world data to validate perception algorithms due to the immense amount of sensor data that needs to be tested. Simulation tools can complement this testing process, as they can fabricate synthetic data based on variable specifications of test cases. While simulative tools can produce a vast amount of test data, at some point, the testing process is limited by the available computing resources. Identifying test specifications that pose risks to the perception algorithms is crucial for efficiently utilizing these computing resources and estimating functional reliability. We present a pipeline for adaptive test case selection to expose the faults of a deep learning system using synthetic image data generated by a simulation framework. We apply our concept to a state-of-the-art object detector and implement multiple adaptive sampling strategies to demonstrate their ability of early fault detection. Our experiments show that our pipeline can achieve a 95% coverage of system faults while reducing the number of test executions by 90%.
Multi-Level Adaptive Separable Convolution for Large-Motion Video Frame Interpolation

Current state-of-the-art methods within Video Frame Interpolation (VI) fail at synthesizing interpolated frames in certain problem areas, such as when the video contains large motion. This work aims at improving performance on frame sequences containing large displacements by extending the Adaptive Separable Convolution model in two ways. First of all, we increase the receptive field of the model by utilizing spatial pyramids, which efficiently increase the interpolation kernel size. We additionally adapt the network to accommodate for four frames, as opposed to just two, which should give it the ability to learn more complex motion patterns. This work also introduces the Large-Motion Video Interpolation Dataset (LMD), which contains extracted frames from videos containing large displacements and highly non-linear movements. Our analysis shows that applying the model changes, together with the use of our new dataset, does indeed result in improved performance on large displacement videos. We also show that the increase in performance generalizes to frame sequences of all sorts by outperforming other models in our benchmark on most tasks, and almost setting the new state-of-the-art on the Vimeo-90K dataset.
Self-Organizing Maps for Cellular In Silico Staining and Cell Substate Classification

Cellular composition and structural organization of cells in the tissue determine effective antitumor response and can predict patient outcome and therapy response. Here we present Seg-SOM, a method for dimensionality reduction of cell morphology in H&E-stained tissue images. Seg-SOM resolves cellular tissue heterogeneity and reveals complex tissue architecture. We leverage a self-organizing map (SOM) artificial neural network to group cells based on morphological features like shape and size. Seg-SOM allows for cell segmentation, systematic classification, and in silico cell labeling. We apply the Seg-SOM to a dataset of breast cancer progression images and find that clustering of SOM classes reveals groups of cells corresponding to fibroblasts, epithelial cells, and lymphocytes. We show that labeling the Lymphocyte SOM class on the breast tissue images accurately estimates lymphocytic infiltration. We further demonstrate how to use Seq-SOM in combination with non-negative matrix factorization to statistically describe the interaction of cell subtypes and use the interaction information as highly interpretable features for a histological classifier. Our work provides a framework for use of SOM in human pathology to resolve cellular composition of complex human tissues. We provide a python implementation and an easy-to-use docker deployment, enabling researchers to effortlessly featurize digitalized H&E-stained tissue.
Feature-Oriented Principal Component Selection (FPCS) for Delineation of the Geological Units Using the Integration of SWIR and TIR ASTER Data

Geological studies have been performed using the Band Ratios (BR), Relative Band Depth (RBD), Mineral Indices (MI), Principal Component Analysis (PCA), Independent Component Analysis (ICA), lithological and mineral classification techniques from Short-Wave Infrared (SWIR) and Thermal Infrared (TIR) data. The chapter aims to delineate various geological units present in the area using the combination of SWIR and TIR ASTER bands through the Feature-Oriented Principal Component Selection (FPCS) technique. Different BRs and RBDs were applied to map the minerals having Al-OH and Mg-OH compounds with the chemical composition of clay (kaolinite, smectite), mica (sericite, muscovite, illite), ultramafic (lizardite, antigorite, chrysotile), talc, and carbonate (dolomite) from SWIR bands. The MI was used to map quartz-rich, mafic/ultramafic, and carbonate rocks using TIR bands. The BRs, RBDs, and MIs mapped the geological units but every single greyscale image showed a variety of features. To compile these features False Color Composite (FCC) was prepared by the combination of RBDs and MIs in the R:G:B channels which demarked various geological units to a larger extent present in the region. To overcome the limitation, the FPCS technique was applied with the integration of all BRs, RBDs, and MIs. The FPCS technique extracts valuable information from different input bands and shifts the information in the first few bands. The generated eigenvalues and eigenvectors represented the retrieved information in the specific band. The loadings of the eigenvector were used for the selection of the different brands to create the FCC for the delineation of geological strata. The best discrimination was made by the selection of FPCS1, FPCS3, and FPCS6 which differentiated all the geological units like ultramafics, dolomites, thin bands of talc, and muscovite and illite (as phyllite and mica-schist), silica-rich rocks (as quartzite), and granite outcrops.
Efficient data hiding method for videos based on adaptive inverted LSB332 and secure frame selection with enhanced Vigenere cipher

In recent years, the importance of information security has increased due to the growth in digital communication and steganography has come to the fore for secure communication. Steganography ensures that the secret data to be sent in a multimedia file such as text, picture or video embedded in a way that cannot be understood by third parties. Although image steganography has become popular recently, video steganography has also been widely used and preferred recently, due to its high data hiding capacity and robustness. This paper proposes a new data hiding algorithm in video steganography for effective frame selection, high embedding capacity and good visual quality. The proposed algorithm has two phases: frame selection and data hiding. In the first phase, the frame selection offers a rapid and impressive new method based on enhanced Vigenere. The most significant advantage of this algorithm is to achieve the same level of security without using complex operations such as chaotic systems used in existing methods. In the second phase, we have used adaptive LSB332 for data hiding, which embeds video files with high capacity, high PSNR and SSIM. When the experimental results evaluated, it was observed that the proposed method is over 65 dB PSNR value and 0.99 SSIM value at 20 kbits data capacity. It was also achieved approximately 55 dB and 50 dB PSNR value at 200 kbits and 500 kbits data capacities, respectively, where is no appreciable loss in the SSIM value. Compared to LSB332 based methods, the capacity increased by approximately 12% at same visual qualities. Furthermore, the visual quality of the proposed method is much better than other video data hiding methods, while the capacity increases by 19%.
A smarthome conversational agent performing implicit demand-response application planning

In recent years, the growing use of Intelligent Personal Agents in different human activities and in various domains led the corresponding research to focus on the design and development of agents that are not limited to interaction with humans and execution of simple tasks. The latest research efforts have introduced Intelligent Personal Agents that utilize Natural Language Understanding (NLU) modules and Machine Learning (ML) techniques in order to have complex dialogues with humans, execute complex plans of actions and effectively control smart devices. To this aim, this article introduces the second generation of the CERTH Intelligent Personal Agent (CIPA) which is based on the RASA framework and utilizes two machine learning models for NLU and dialogue flow classification. CIPA-Generation B provides a dialogue-story generator that is based on the idea of adjacency pairs and multiple intents, that are classifying complex sentences consisting of two users’ intents into two automatic operations. More importantly, the agent can form a plan of actions for implicit Demand-Response and execute it, based on the user’s request and by utilizing AI Planning methods. The introduced CIPA-Generation B has been deployed and tested in a real-world scenario at Centre’s of Research &amp; Technology Hellas (CERTH) nZEB SmartHome in two different domains, energy and health, for multiple intent recognition and dialogue handling. Furthermore, in the energy domain, a scenario that demonstrates how the agent solves an implicit Demand-Response problem has been applied and evaluated. An experimental study with 36 participants further illustrates the usefulness and acceptance of the developed conversational agent-based system.
LeARn: A Collaborative Learning Environment using Augmented Reality

This work presents “LeARn”, a new network-based collaborative learning environment that employs augmented reality to transform a real-world surface in a virtual lab. The system is a contribution towards replacing a face-to-face learning environment with an augmented collaborative setting. To demonstrate the system, a scenario with a virtual chemistry lab is presented. In this demo, any real-world surface is augmented by virtual lab equipment utilized in a chemistry experiment. The virtual lab is hosted by the instructor, and all the students can join the lab only using their mobile phones or tablets. Each member can interact with the lab equipment, which can be visualized in real-time by the instructor or fellow students. The system allows for real-time communication that fosters a truly collaborative environment. The resulting system demonstrates that a complex lab experiment can be performed from a personalized space that can incorporate positive traits of a collaborative environment. The system was deployed and evaluated in an uncontrolled user study, and the results show the effectiveness of an AR-based interactive and collaborative learning environment.
Assessment in the zone of proximal development: simulator-based competence tests and the dynamic evaluation of knowledge-in-action

The background of this article is an interest in analysing how assessment of professional skills is conducted in higher education contexts, drawing on video data from a course on maritime navigation. The empirical study focuses on a) how students working in a bridge simulator are able to display their knowledge about how to calculate the relation between rate of turn, turn rate and speed when navigating, and b) how their performance is evaluated. The results show that the dynamics of the assessment situation and the interaction between the student and the assessor are decisive in how well students manage the task. When analysing assessment of a professional skill as a concrete practice, knowledge emerges through joint work and cannot be ascribed to the student only. At one level, this may be seen as a threat to the validity of the testing situation, but from a different perspective, the display of knowledge-in-action is always responsive to what happens in the exercise of a professional skill.
Effect of Parameter Optimization on Classical and Learning-based Image Matching Methods

Deep learning-based image matching methods are improved significantly during the recent years. Although these methods are reported to outperform the classical techniques, the performance of the classical methods is not examined in detail. In this study, we compare classical and learning-based methods by employing mutual nearest neighbor search with ratio test and optimizing the ratio test threshold to achieve the best performance on two different performance metrics. After a fair comparison, the experimental results on HPatches dataset reveal that the performance gap between classical and learning-based methods is not that significant. Throughout the experiments, we demonstrated that SuperGlue is the state-of-the-art technique for the image matching problem on HPatches dataset. However, if a single parameter, namely ratio test threshold, is carefully optimized, a well-known traditional method SIFT performs quite close to SuperGlue and even outperforms in terms of mean matching accuracy (MMA) under 1 and 2 pixel thresholds. Moreover, a recent approach, DFM, which only uses pre-trained VGG features as descriptors and ratio test, is shown to outperform most of the well-trained learning-based methods. Therefore, we conclude that the parameters of any classical method should be analyzed carefully before comparing against a learning-based technique.
An Effective Fair Off-Line Electronic Cash Protocol using Extended Chaotic Maps with Anonymity Revoking Trustee

Along with the proliferation of cyberspace and the beginning of electronic trade, many electronic cash protocols were suggested. Electronic cash allows digital coins to be exchanged with value guaranteed by the signature of the financial institution and the hidden identity of the client. A client can withdraw money from the financial institution in an electronic cash protocol and then anonymously and unlinkably spend each coin. The present article suggests a practical, fair offline electronic cash protocol using extended chaotic maps capable of coin locating and seller locating. Under certain conditions, the protocol’s anonymity perchance was revoked from an offline trusted third party. The trustworthy third party verifies the financial institution’s e-coin signature in our protocol and then logs the location of data that isn’t part of the normal electronic cash protocol.
Exploiting Verified Neural Networks via Floating Point Numerical Error

Researchers have developed neural network verification algorithms motivated by the need to characterize the robustness of deep neural networks. The verifiers aspire to answer whether a neural network guarantees certain properties with respect to all inputs in a space. However, many verifiers inaccurately model floating point arithmetic but do not thoroughly discuss the consequences.
Causal-Aware Generative Imputation for Automated Underwriting

Underwriting is an important process in insurance and is concerned with accepting individuals into insurance policy with tolerable claim risk. Underwriting is a tedious and labor intensive process relying on underwriters' domain knowledge and experience, thus is labor intensive and prone to error. Machine learning models are recently applied to automate the underwriting process and thus to ease the burden on the underwriters as well as improve underwriting accuracy. However, observational data used for underwriting modelling is high dimensional, sparse and incomplete, due to the dynamic evolving nature (e.g., upgrade) of business information systems. Simply applying traditional supervised learning methods e.g., logistic regression or Gradient boosting on such highly incomplete data usually leads to the unsatisfactory underwriting result, thus requiring practical data imputation for training quality improvement. In this paper, rather than choosing off-the-shelf solutions tackling the complex data missing problem, we propose an innovative Generative Adversarial Nets (GAN) framework that can capture the missing pattern from a causal perspective. Specifically, we design a structural causal model to learn the causal relations underlying the missing pattern of data. Then, we devise a Causality-aware Generative network (CaGen) using the learned causal relationship prior to generating missing values, and correct the imputed values via the adversarial learning. We also show that CaGen significantly improves the underwriting prediction in real-world insurance applications.