Reviewerly/exHarmony huggingface dataset:
Multi-Modal Contrastive Pre-training for Recommendation

Personalized recommendation plays a central role in various online applications. To provide quality recommendation service, it is of crucial importance to consider multi-modal information associated with users and items, e.g., review text, description text, and images. However, many existing approaches do not fully explore and fuse multiple modalities. To address this problem, we propose a multi-modal contrastive pre-training model for recommendation. We first construct a homogeneous item graph and a user graph based on the relationship of co-interaction. For users, we propose intra-modal aggregation and inter-modal aggregation to fuse review texts and the structural information of the user graph. For items, we consider three modalities: description text, images, and item graph. Moreover, the description text and image complement each other for the same item. One of them can be used as promising supervision for the other. Therefore, to capture this signal and better exploit the potential correlation of intra-modalities, we propose a self-supervised contrastive inter-modal alignment task to make the textual and visual modalities as similar as possible. Then, we apply inter-modal aggregation to obtain the multi-modal representation of items. Next, we employ a binary cross-entropy loss function to capture the potential correlation between users and items. Finally, we fine-tune the pre-trained multi-modal representations using an existing recommendation model. We have performed extensive experiments on three real-world datasets. Experimental results verify the rationality and effectiveness of the proposed method.
Non-homogeneous haze data synthesis based real-world image dehazing with enhancement-and-restoration fused CNNs

Single image dehazing is a challenging ill-posed task, we address it from the aspects of dehazing dataset and network architecture. As for the dehazing dataset, there are such problems as unnatural hazy images, unqualified ground truths, as well as monotonous and idealized depth-related haze synthesized by the physical model in the existing dehazing datasets. Therefore, we propose a novel haze data synthesis method to produce a dehazing dataset with non-homogeneous haze, which is named as FiveK-Haze. As for the network architecture, existing methods either get the image enhancement results directly in an end-to-end approach or restore the haze-free images based on the estimated physical parameters in a multiple-branch approach. To get better results for the real-world non-homogeneous haze images, we combine above two approaches together in a complementary way and design a new dehazing network with the enhancement-and-restoration fused CNNs, which is called as ERFNet. Exhaustive experimental results demonstrate the superiority of our method over the state-of-the-art methods in terms of both generalization performance and haze removal effects, especially for detail enhancement and color restoration.
MEMORIA: A Memory Enhancement and MOment RetrIeval Application for LSC 2022

Research on retrieving data and analyzing lifelogs revealed to be a very complex task, and the interdisciplinary challenges to be tackled have boosted increasing attention from the scientific community in information retrieval and lifelogging. The Lifelog Search Challenge is an international competition for lifelog retrieval in which researchers propose their approaches and compete to solve lifelog retrieval challenges and evaluate the effectiveness of their systems. In this paper, we present the MEMORIA computational tool to participate for the first time in the Lifelog Search Challenge 2022. The information retrieval is based on the search of keywords and time periods and several computer vision methods are used to process visual lifelogs, from pre-processing algorithms to feature extraction methods, in order to enrich the annotation of the lifelogs. Preliminary experimental results of the user interaction with our retrieval module are presented, confirming the effectiveness of the proposed approach and showing the most relevant functionalities of the system.
AutoTinyML for microcontrollers: Dealing with black-box deployability

While many companies are currently leveraging on Cloud, data centres and specialized hardware (e.g., GPUs and TPUs) to train very accurate Machine Learning models, the need to deploy and run these models on tiny devices is emerging as the most relevant challenge, with a massive untapped market. Although Automated Machine Learning and Neural Architecture Search frameworks are successfully used to find accurate models by trying a small number of alternatives, they are typically performed on large computational platforms and they cannot directly deal with deployability, leading to an accurate model which could result undeployable on a tiny device. To bridge the gap between these two worlds, we present an approach extending these frameworks to include the constraints related to the limited hardware resources of the tiny device which the trained model has to run on. Experimental results on two benchmark classification tasks and two microcontrollers prove that our AutoTinyML framework can efficiently identify models which are both accurate and deployable, in case accepting a reasonable reduction in accuracy compared to a significant reduction in hardware usages, without applying any quantization techniques of the model.
Medical Gesture Recognition Method Based on Improved Lightweight Network

Surgery is a compelling application field for collaborative control robots. This paper proposes a gesture recognition method applied to a medical assistant robot delivering instruments to collaborate with surgeons to complete surgeries. The key to assisting the surgeon in passing instruments in the operating room is the ability to recognize the surgeon’s hand gestures accurately and quickly. Existing gesture recognition techniques suffer from poor recognition accuracy and low rate. To address the existing shortcomings, we propose an improved lightweight convolutional neural network called E-MobileNetv2. The ECA module is added to the original MobileNetv2 network model to obtain more useful features by computing the information interactions between the current channel and the adjacent channels and between the current channel and the distant channels in the feature map. We add R6-SELU activation function to enhance the network’s ability to extract features. By adjusting the shrinkable hyper-parameters, the number of parameters of the network is reduced to improve the recognition speed. The improved network model achieves excellent performance on both the self-built dataset Gesture_II and the public dataset Jester. The recognition accuracy of the improved model is 96.82%, which is 3.17 % higher than that of the original model, achieving an increase in accuracy and recognition speed.
Sophistication with Limitation: Understanding Smartphone Usage by Emergent Users in India

India has been witnessing a steady increase in smartphone penetration since 2016 after Reliance Jio introduced inexpensive internet plans. Much of HCI research in the Global South has been conducted before smartphones became more widespread. More recent work on smartphone use in India has been either domain-focused or studied specific features. In this work, we investigate how emergent users from low-income communities in India currently use their smartphones, and what they use them for. We draw on semi-structured interviews with emergent smartphone users across rural and urban India demonstrating their experiences and challenges related to low- textual and digital literacy, infrastructure, privacy, and motivations of use. Our findings revealed that while there is a lack of understanding of basic features such as accounts and passwords, there is sophisticated use spanning user-generated media, remote education, skilling, etc. We close with recommendations for future research and design for emergent smartphone users.
Predicting the Mortality of ICU Patients by Topic Model with Machine-Learning Techniques

Predicting clinical patients’ vital signs is a leading critical issue in intensive care units (ICUs) related studies. Early prediction of the mortality of ICU patients can reduce the overall mortality and cost of complication treatment. Some studies have predicted mortality based on electronic health record (EHR) data by using machine learning models. However, the semi-structured data (i.e., patients’ diagnosis data and inspection reports) is rarely used in these models. This study utilized data from the Medical Information Mart for Intensive Care III. We used a Latent Dirichlet Allocation (LDA) model to classify text in the semi-structured data of some particular topics and established and compared the classification and regression trees (CART), logistic regression (LR), multivariate adaptive regression splines (MARS), random forest (RF), and gradient boosting (GB). A total of 46,520 ICU Patients were included, with 11.5% mortality in the Medical Information Mart for Intensive Care III group. Our results revealed that the semi-structured data (diagnosis data and inspection reports) of ICU patients contain useful information that can assist clinical doctors in making critical clinical decisions. In addition, in our comparison of five machine learning models (CART, LR, MARS, RF, and GB), the GB model showed the best performance with the highest area under the receiver operating characteristic curve (AUROC) (0.9280), specificity (93.16%), and sensitivity (83.25%). The RF, LR, and MARS models showed better performance (AUROC are 0.9096, 0.8987, and 0.8935, respectively) than the CART (0.8511). The GB model showed better performance than other machine learning models (CART, LR, MARS, and RF) in predicting the mortality of patients in the intensive care unit. The analysis results could be used to develop a clinically useful decision support system.
On the Characteristics of Ranking-based Gender Bias Measures

With increased recent awareness on the possible impact of retrieval techniques on intensifying gender biases, researchers have embarked on defining quantifiable gender bias metrics that can provide the means to concretely measure such biases in practice. While successful in allowing for identifying possible sources of gender bias, there has been little work that systematically explores the characteristics of these metrics. This paper argues that effective future works on gender biases in information retrieval require a careful understanding of the bias metrics in terms of their consistency, robustness, sensitivity and also their relation with psychological characteristics and what they actually measure. Through our experiments, we show that more rigorous work on gender bias metrics need to be pursued as existing metrics may not necessarily be consistent and robust and often capture differing psychological characteristics.
Optimal energy efficient path planning of UAV using hybrid MACO-MEA* algorithm: theoretical and experimental approach

Autonomous mission capabilities with optimal path are stringent requirements for Unmanned Aerial Vehicle (UAV) navigation in diverse applications. The proposed research framework is to identify an energy-efficient optimal path to achieve the designated missions for the navigation of UAVs in various constrained and denser obstacle prone regions. Hence, the present work is aimed to develop an optimal energy-efficient path planning algorithm through combining well known modified ant colony optimization algorithm (MACO) and a variant of A*, namely the memory-efficient A* algorithm (MEA*) for avoiding the obstacles in three dimensional (3D) environment and arrive at an optimal path with minimal energy consumption. The novelty of the proposed method relies on integrating the above two efficient algorithms to optimize the UAV path planning task. The basic design of this study is, that by utilizing an improved version of the pheromone strategy in MACO, the local trap and premature convergence are minimized, and also an optimal path is found by means of reward and penalty mechanism. The sole notion of integrating the MEA* algorithm arises from the fact that it is essential to overcome the stringent memory requirement of conventional A* algorithm and to resolve the issue of tracking only the edges of the grids. Combining the competencies of MACO and MEA*, a hybrid algorithm is proposed to avoid obstacles and find an efficient path. Simulation studies are performed by varying the number of obstacles in a 3D domain. The real-time flight trials are conducted experimentally using a UAV by implementing the attained optimal path. A comparison of the total energy consumption of UAV with theoretical analysis is accomplished. The significant finding of this study is that, the MACO-MEA* algorithm achieved 21% less energy consumption and 55% shorter execution time than the MACO-A*. moreover, the path traversed in both simulation and experimental methods is 99% coherent with each other. it confirms that the developed hybrid MACO-MEA* energy-efficient algorithm is a viable solution for UAV navigation in 3D obstacles prone regions.
A Probabilistic View on Rupture Predictability: All Earthquakes Evolve Similarly

Ruptures of the largest earthquakes can last between a few seconds and several minutes. An early assessment of the final earthquake size is essential for early warning systems. However, it is still unclear when in the rupture history this final size can be predicted. Here we introduce a probabilistic view of rupture evolution - how likely is the event to become large - allowing for a clear and well-founded answer with implications for earthquake physics and early warning. We apply our approach to real time magnitude estimation based on either moment rate functions or broadband teleseismic P arrivals. In both cases, we find strong and principled evidence against early rupture predictability because differentiation between differently sized ruptures only occurs once half of the rupture has been observed. Even then, it is impossible to foresee future asperities. Our results hint towards a universal initiation behavior for small and large ruptures.
Hostility Detection in Online Hindi-English Code-Mixed Conversations

With the rise in accessibility and popularity of various social media platforms, people have started expressing and communicating their ideas, opinions, and interests online. While these platforms are active sources of entertainment and idea-sharing, they also attract hostile and offensive content equally. Identification of hostile posts is an essential and challenging task. In particular, Hindi-English Code-Mixed online posts of conversational nature (which have a hierarchy of posts, comments, and replies) have escalated the challenges. There are two major challenges: (1) the complex structure of Code-Mixed text and (2) filtering the relevant previous context for a given utterance. To overcome these challenges, in this paper, we propose a novel hierarchical neural network architecture to identify hostile posts/comments/replies in online Hindi-English Code-Mixed conversations. We leverage large multilingual pre-trained (mLPT) models like mBERT, XLMR, and MuRIL. The mLPT models provide a rich representation of code-mix text and hierarchical modeling leads to a natural abstraction and selection of the relevant context. The propose model consistently outperformed all the baselines and emerged as a state-of-the-art performing model. We conducted multiple analyses and ablation studies to prove the robustness of the proposed model.
Low complexity template-based watermarking with neural networks and various embedding templates

The popularity of watermarking was amplified due to the emergence of new immersive applications that connect the digital and physical worlds such as reading with a smartphone a watermark located in a physical object that leads to an online, dynamic source. Such applications require new watermarking schemes which must be robust enough to withstand significant watermark distortions and have low complexity in the sense that they must be adapted to be used on devices with constrained resources that are common in today's cyber-physical world. The key contribution of this work is the novel robust and low-complex template-based watermarking scheme that uses neural networks. Neural networks have been applied as they have already shown a distinguishable result in such image processing tasks as object detection and image restoration. The experiments show that our scheme outperforms the existing state-of-the-art template-based schemes in terms of robustness without visual degradation.
Conceptual Design of an Extended Reality Exercise Game for the Elderly

Facing the increasingly serious ageing problem, IT technology is being applied more and more to alleviate the increasing health demands of the elderly. Virtual reality (VR) games, in particular, have been shown to benefit the physical and mental health of the elderly. However, almost all consumer-level VR games address the general user, not considering the characteristics of the elderly, such as their particular needs regarding cognitive aspects and safety. In addition, the existing VR rowing games’ research typically focuses on athlete training or is based on a rowing machine. Extending on previous research, this paper proposes a conceptual design of a VR rowing game, VRrowing to provide a home exercise game for the elderly which benefits both physical and mental health. This research first analysed the relevant literature and existing VR exercise games for the elderly to find characteristics and their particular needs. Then, following design principles (in exercise games, human–computer interaction, and user interfaces (UIs)) for the elderly, this study proposed a conceptual design of VRrowing through the MDA framework (standing for mechanics, dynamics, and aesthetics). The results cover the system architecture, activities and tasks, patterns and roles, mechanics and structures, and UI examples. Based on comprehensive consideration of the results from relevant research and fostering strengths and circumventing weaknesses, VRrowing was designed as a sports game, aimed at the elderly according to their characteristics and needs, which fills the void of home VR rowing games for seniors’ exercise. This research has reference value for scientific research or development personnel in VR game applications and evaluations, especially for such games targeting the elderly.
Data privacy protection in scientific publications: process implementation at a pharmaceutical company

Sharing anonymized/de-identified clinical trial data and publishing research outcomes in scientific journals, or presenting them at conferences, is key to data-driven scientific exchange. However, when data from scientific publications are linked to other publicly available personal information, the risk of reidentification of trial participants increases, raising privacy concerns. Therefore, we defined a set of criteria allowing us to determine and minimize the risk of data reidentification. We also implemented a review process at Takeda for clinical publications prior to submission for publication in journals or presentation at medical conferences.Abstracts, manuscripts, posters, and oral presentations containing study participant information were reviewed and the potential impact on study participant privacy was assessed. Our focus was on direct (participant ID, initials) and indirect identifiers, such as sex, age or geographical indicators in rare disease studies or studies with small sample size treatment groups. Risk minimization was sought using a generalized presentation of identifier-relevant information and decision-making on data sharing for further research. Additional risk identification was performed based on study participant/personnel parameters present in materials destined for the public domain. The potential for participant/personnel identification was then calculated to facilitate presentation of meaningful but de-identified information.The potential for reidentification was calculated using a risk ratio of the exposed versus available individuals, with a value above the threshold of 0.09 deemed an unacceptable level of reidentification risk. We found that in 13% of Takeda clinical trial publications reviewed, either individuals could potentially be reidentified (despite the use of anonymized data sets) or inappropriate data sharing plans could pose a data privacy risk to study participants. In 1/110 abstracts, 58/275 manuscripts, 5/87 posters and 3/58 presentations, changes were necessary due to data privacy concerns/rules. Despite the implementation of risk-minimization measures prior to release, direct and indirect identifiers were found in 11% and 34% of the analysed documents, respectively.Risk minimization using de-identification of clinical trial data presented in scientific publications and controlled data sharing conditions improved privacy protection for study participants. Our results also suggest that additional safeguards should be implemented to ensure that higher data privacy standards are met.
GNoM: Graph Neural Network Enhanced Language Models for Disaster Related Multilingual Text Classification

Online social media works as a source of various valuable and actionable information during disasters. These information might be available in multiple languages due to the nature of user generated content. An effective system to automatically identify and categorize these actionable information should be capable to handle multiple languages and under limited supervision. However, existing works mostly focus on English language only with the assumption that sufficient labeled data is available. To overcome these challenges, we propose a multilingual disaster related text classification system which is capable to work undervmonolingual, cross-lingual and multilingual lingual scenarios and under limited supervision. Our end-to-end trainable framework combines the versatility of graph neural networks, by applying over the corpus, with the power of transformer based large language models, over examples, with the help of cross-attention between the two. We evaluate our framework over total nine English, Non-English and monolingual datasets invmonolingual, cross-lingual and multilingual lingual classification scenarios. Our framework outperforms state-of-the-art models in disaster domain and multilingual BERT baseline in terms of Weighted F1 score. We also show the generalizability of the proposed model under limited supervision.