Reviewerly/exHarmony huggingface dataset:
Image enhancement for dichromats using image pyramid based on saturation

Dichromats recognize colors using two out of three cone cells; L, M, and S. For example, red-green color blinds can-not distinguish the color between red, yellow, and green. To extend the ability of dichromats to recognize the color dif-ference, we propose a method to expand the color difference when observed by dichromats. We analyze the color between the neighboring pixels in chromaticity space. In addition, we employ multiresolution analysis to form the Pois-son equation. Our multiresolution analysis is non-linear depending on the saturation of each pixel's color. Solving the multiresolution Poisson equation results in the color en-hanced image.
Recursive Randomized Tree Coding of Speech

We study a recursively adaptive architecture for speech coding based on the concept of tree coding combined with recursive least squares lattice estimation of the autoregressive component and gradient based estimation of the moving average part of the short term prediction and gradient/autocorrelation based long term prediction algorithms, all adapting to minimize the perceptually weighted reconstruction error. The new idea of concatenated, randomized multitrees is introduced and explored. Voice activity detection (VAD) and comfort noise generation (CNG) are included to reduce the bit rate and the number of computations required. Performance is compared to the widely implemented and utilized AMR codec and we demonstrate comparable performance at bit rates of 4.5 to 7.5 kbits/s.
ImpartialGAN: Fair and Unbiased Classification

In this age of big data, one of the key concerns in the recent days has been bias present in the data and hence the need to ensure data fairness. There is a need to ensure that bias in the data does not reflect in the models decision which in turn treats people from certain race, gender, sexual or political orientation unfairly and differently. The goal of fair data generation is to remove any prejudice which might be present in the data towards any specific demographic group. This is particularly of interest in decision making scenarios like financial lending, hiring, pretrial and immigration detention, health care, social services, and education where the system might favor one race and is biased towards the other. In this paper, we propose ImpartialGAN to generate fair synthetic data from real data. The generated data is not only fair and free from bias but also ensures a good data utility while preserving data privacy. Hence this generated data can be used in place of real data for predictive analytics. In our experiments on UCI Adult dataset, we achieved 83.43 % accuracy on real data while keeping the risk difference for synthetic data at 0.0063, which indicates that our classifier is fair and unbiased.
Counting Pixels for an Effective Axis Detection

To examine the charts critically and to synthesize the charts across relevant papers in a large-scale accessible digitally, we need an efficient way to extract data from visual information in the charts. Axis detection is the crux in such data extraction from the charts. Thus, we introduce a rule-based axis detection algorithm that is computationally less expensive and more accurate by using simple pixel counting.
Work-In-Progress—Exploring the Feasibility of Using Hand Tracking in VR Application for Memory Training Exercises

There are several non-pharmaceutical treatment approaches, from simple to more complex, for the care of dementia patients, such as the use of immersive technology with virtual reality (VR). However, there are some difficulties in using wired and controller-based VR devices for dementia patients. This work-in-progress paper describes our steps to implement a wireless VR experience with a hand-tracking interaction system. Before testing the environment with patients, we first evaluate the usability of the VR experience by conducting a user study with ten participants. The results indicate above-average usability and low task load. Future developments in this area may improve the application of VR in memory training.
Learning Rotational Invariant Dictionary for Sparse Coding based Key-point Detection

Recent research has shown the effectiveness of a Scale and Rotational Invariant Sparse Coding based Key-point Detector. However, this detector utilizes a dictionary generated by combining a carefully selected seed dictionary with multiple rotated versions of its atoms. This manual generation process is time-demanding, thus a novel autoencoder structure for automating it is proposed. Specifically, the weights between input and hidden layers of the structure are designed to embed a rotational invariant dictionary. Indeed, it is a duplet structure constrained by a novel loss function such that rotated inputs correspond to circularly shifted versions of hidden layer responses. To our knowledge, an autoencoder embedded in its weight a rotational invariant dictionary is new and the proposed loss function is the first to fulfill this requirement. Experimental results show that the learned dictionary preserves the original detector's performance, helps achieve state-of-the-art key-point detection performance while substantially eliminating the laborious work.
Machine Learning for Self-Calibration Parameters of Data-Driven Models: Case Study of an Integrated Maintenance Digital Platform

A well-established problem always in the process of methodological improvement is the coordination of generator maintenance scheduling in Power Systems. Usually, the sources of changes and improvements are the wide range of conditions and singularities of Power Systems, such as structure, needs, technologies, resources, information, etc. The risk-oriented approach is one of the most accepted criteria to address this well-defined problem. However, there are hidden challenges of data-calibration-modeling and technology-integration when applying this approach. One of the most controversial is the calibration of model parameters and variables in the evaluated scenario, as they depend on historical data. In this paper, to address this controversial hidden challenge, we propose to use machine learning for on-line calibration by introducing smart layers based on comprehensive human diagrams that filter and analyze historical data collected through monitoring systems and make all variables and parameters of the risk model ready for use. In particular, we focus the attention on one of the model components, the load modeling. This proposed organic connection ensures a feasible and practical solution to be used in a real system because it addresses a potential challenge in technology integration.
Framework for Scalable Content Development in Hands-On Virtual and Mixed Reality Science Labs

Authentic hands-on laboratory research is essential for undergraduate STEM education. Yet the tactile authenticity required to impact affective, cognitive, or psychomotor learning outcomes associated with laboratory training remains underexplored. Virtual and mixed reality (VR/MR) have enabled increasingly realistic hands-on STEM training experiences. However, they still lack authenticity with regard to user manipulation of fully functional and realistic laboratory tools, analysis of realistic (i.e. user-acquired) noisy data, and the application of critical thinking skills to draw conclusions from such noisy (and possible faulty) data. Here we present efforts to develop such an approach while also providing faculty content experts tools for code-free customization of VR/MR training experiences via structured spreadsheets. This approach enables nuanced real-time user feedback on laboratory skills such as proper pipetting or sterile technique which are otherwise difficult to provide. It also offers complete safety from chemical, biological, and radiological hazards and is more cost-effective than a traditional lab. This Hands-On Virtual-Reality (HOVR) Lab platform is uniquely enabling and will be valuable in the physical and life sciences for both research and instructional applications.
Improving Sentence Classification in Abstracts of Randomized Controlled Trial using Prompt Learning

We proposed a Prompt Learning (PL) based Sentence Classification method to structure abstracts of Randomized Controlled Trial (RCT) articles. Five manually designed templates and four different BERT models were developed for PL and we evaluated them on a dataset of 20K PubMed RCT abstracts. We also compared them with the hierarchical sequential labeling network (HSLN) approach, which previously achieved the best performance in relevant work. Our results show that the PL-based method outperformed HSLN by 0.0138, achieving an F1 score of 0.9401 on the PubMed 20k dataset, indicating the promise of PL in this task.
Work-in-Progress–Computer Vision Methods to Examine Neurodiverse Gaze Patterns in 360-Video

Computer vision (CV) is a subset of artificial intelligence (AI) that focuses on enabling computers to detect and understand objects from visual stimuli and multimedia. With advances in computational power and open-source libraries, more educators and instructional designers are seeking to capitalize on the perceived benefits of CV. This work-in-progress paper reports the CV approach our research team has developed to explore the usage patterns of neurodiverse learners within a 360-degree spherical video-based virtual reality system.
A Noise-robust Locality Transformer for Fine-grained Food Image Retrieval

Food image retrieval has wide applications in the multimedia community. However, there are two main challenges for food image retrieval. First, food images are often disturbed by food-irrelevant information such as plates and side dishes. Second, fine-grained characteristics of food images make visual representation of different categories similar. To solve them, we propose the Noiserobust Locality Transformer (NoLoTransformer) for food image retrieval under the metric learning-based retrieval framework. Specifically, we propose two novel modules, named Patch Attention Module (PAM) and Local Perception Unit (LPU) for the Transformer-based feature extraction. PAM weakens the negative impact of the noise in the food image by reweighting different patches and distributing low weights to noisy patches adaptively. LPU extracts local features by introducing convolution and then obtains the fine-grained information in the local feature. Extensive evaluation on three datasets demonstrates the effectiveness of the proposed method. Code is available at https://github.com/jiajun-ISIA/NoLoTransformer
Study of Adversarial Machine Learning for Enhancing Human Action Classification

Action classification has traditionally been an effective way of identifying threatening and nonthreatening targets with significant traffic in military and civilian locations. Motivated actors in place of accessing detailed model requirements can simulate adversarial attacks on an action recognition networks that are highly transferable to production models. The problem is that adversarial attacks against spatial-temporal data are straightforward to generate since they are sequential and hard to detect. We propose using maximal informative adversarial training to create robust classification using Human action classification networks (HCNs). Using adversarial samples of the real action data, we can create synthetic sequences that expand the range of motions our classifier expects for a given class of actions. Our research demonstrates the effectiveness of adversarial trained HCNs against the transferred adversarial attacks. Our method of adversarial training using maximum informativeness outperformed defense by inference when comparing models that were transfer attacked using sequences of motions that are effective on previous models. We find that non-contextual human action recognition architectures are more susceptible than those using context from the scenes they are analyzing. Our contextual model performed 47 percent better after adversarial training. Contextual followed by graph-based models had the lowest attack success rate of all surrogate models trained.
Risk-Based Breast Cancer Prognosis Using Minimal Patient Characteristics

The use of machine learning and deep learning for cancer prediction has been proven to be relevant and promising in the past. In this work, we propose a machine learning modeling approach for breast cancer prognosis with primary information about the user, such as their age and race. One of the essential aspects of machine learning application to the health domain is the ability to robustly handle the uneven distribution of output classes in the data using resampling techniques. In this paper, we present the results before and after applying resampling and observe a significant improvement as a result of handling the imbalance in the data. We employ conventional and contemporary machine learning and deep learning algorithms to predict breast cancer in women and achieve an accuracy of 92.46% on real world datasets.
Steel Delivery Order Recognition Based on Deep Learning and Posterior Error Correction Technology

The main voucher for the circulation of bulk goods is the delivery order. The traditional warehousing enterprise receives the goods through manual input, which is inefficient and error-prone. The recognition rate of the traditional algorithm model is low and cannot be applied on a large scale. According to the characteristics of steel delivery order, through the integration of algorithm technology, this paper proposes an algorithm model based on image correction, text location, text recognition, and post verification, which solves the problem of the low recognition rate of the traditional algorithm. The character recognition rate of the recognition model is more than 95%. Finally, a visual manual correction function is developed to ensure 100% accuracy of output text data. Based on the intelligent identification technology of delivery orders, the traditional goods circulation mode is transformed from the series process of offline circulation based on traditional paper documents to the parallel process with an information-sharing cloud platform as the carrier. We build an intelligent information management system of delivery order of bulk goods is constructed. The business practice shows that the system can quickly and accurately extract the text information of delivery documents and effectively improve the efficiency of goods warehousing and circulation.
Evaluation of Document-Level Identification of Pulmonary Nodules in Radiology Reports Using FLAIR Natural Language Processing Framework

Incidental radiographic findings are commonly found in radiology reports, can have significant clinical implications, yet frequently are missed or not followed-up on due to clinician error. This work evaluates a document-level classification task of radiology reports for a type of incidental finding, pulmonary nodules, utilizing the FLAIR NLP framework as a proof-of-concept for potential automation of identification of such findings for more consistent tracking.
Reduce the Cold Start of COVID-19 In-hospital Mortality Prediction Models via Transfer Learning

At the beginning of the breakout of a new disease, the healthcare community almost always has little experience in treating patients of this kind. Similarly, due to insufficient patient records at the early stage of a pandemic, it is difficult to train an in-hospital mortality prediction model specific to the new disease. We call this the “cold start” problem of mortality prediction models. In this paper, we aim to study the cold start problem of 3-days ahead COVID-19 mortality prediction models by the following two steps: (i) Train XGBoost [1] and logistic regression 3-days ahead mortality prediction models on MIMIC3, a publicly available ICU patient dataset [2];(ii) Apply those MIMIC3 models to COVID-19 patients and then use the prediction scores as a new feature to train COVID-19 3-days ahead mortality prediction models. Retrospective experiments are conducted on a real-world COVID-19 patient dataset(n = 1,287) collected in US from June 2020 to February 2021 with a mixed cohort of both ICU and Non-ICU patients. Since the dataset is imbalanced(death rate = 7.8%), we primarily focus on the relative improvement of AUPR. We trained models with and without MIMIC3 scores on the first 200, 400,…, 1000 patients respectively and then tested on the next 200 incoming patients. The results show a diminishing positive transfer effect of AUPR from 5.36% for the first 200 patients(death rate = 5.5%) to 3.58% for all 1,287 patients. Meanwhile the AUROC scores largely remain unchanged, regardless of the number of patients in the training set. What's more, the p-value of t-test suggests that the cold start problem disappears for a dataset larger than 600 COVID-19 patients. To conclude, we demonstrate the possibility of mitigating the cold start problem via the proposed method.
D<sup>2</sup>-UTransforIner: Deep Modulated Dual-UTransformer for Multiple Description Image Enhancement

Different degrees of distortions exist for the side-and center-compressed images during multiple description (MD) coding. To resolve this problem, we propose an MD image enhancement method, namely, deep modulated dual-UTransformer, which combines Convolutional Neural Network (CNN) with transformer in terms of their locality and global properties. Firstly, we design dualncoder network to extract complementary features and increase feature di-versity at each scale, and dualdecoder network shares the same motivation. Secondly, we introduce a kind of MultiPath Transformer (MPT) layer. Besides, we design a modulation network to adjust dualdecoder network to achieve adaptive image enhancement for side- and center-decoded images. Extensive experimental results have demonstrated that the proposed method is superior to many CNN-based image enhancement methods as well as Uformer approach.
Automatic Evaluation of Machine Generated Feedback For Text and Image Data

In this paper, a novel system, ‘AutoEvaINet,’ has been developed for evaluating machine-generated feedback in response to multimodal input containing text and images. A new metric, ‘Automatically Evaluated Relevance Score’ (AER Score), has also been defined to automatically compute the similarity between human-generated comments and machine-generatedfeedback. The AutoEvalNet's architecture comprises a pre-trained feedback synthesis model and the proposed feedback evaluation model. It uses an ensemble of Bidirectional Encoder Representations from Transformers (BERT) and Global Vectors for Word Representation (GloVe) models to generate the embeddings of the ground-truth comment and machine-synthesized feedback using which the similarity score is calculated. The experiments have been performed on the MMFeed dataset. The generated feedback has been evaluated automatically using the AER score and manually by having the human users evaluate the feedbackfor relevance to the input and ground-truth comments. The values of the AER score and human evaluation scores are in line, affirming the AER score's applicability as an automatic evaluation measure for machine-generated text instead of human evaluation.
LIVE: A Local Interpretable model-agnostic Visualizations and Explanations

Deep learning has been successfully applied to various types of classification and prediction tasks including images, text, and tabular data, however interpretability of deep learning models still remains challenges, which becomes a limiting factor for further applications.Most of current interpretable models are limited to how to interpret the deep learning models. In this paper, we propose a simple but effective novel Local Interpretable model-agnostic Visualizations and Explanations (LIVE) algorithm through combining decision tree and deep learning models. The proposed LIVE algorithm is validated through 4 different data sets with different types of deep learning models including convolutional neural network (CNN) and long short-term memory models (LSTM). Our experiment results show that 1) LIVE algorithm could generally consistently help improve the deep learning models by 0.02 0.03 in AUC; 2) Our LIVE algorithm is a weakly supervised localization tool for region of interest identification; 3) Our LIVE algorithm could be applied to different deep learning models and different types of data.
Effects of Information Masking in the Task-Specific Finetuning of a Transformers-Based Clinical Question-Answering Framework

Transformers-based language models have achieved impressive performance in biomedical question-answering (QA). Our previous work led to surmise that such models could leverage frequent literal question-answer pairs to get the correct answers, casting doubt on true intelligence and transferability. Therefore, we conducted experiments by masking the anchor concept in the question and context documents during the fine-tuning stage of BERT for a reading comprehension QA task on clinical notes. The perturbation involved randomly replacing 0%, 10%, 20%, 30%, and 100% of the concept occurrences into a dummy string. We found the 100% masking harshly penalized the overall accuracy by about 0.10 versus 0% masking. However, the accuracy improved about 0.01 to 0.02 at 20% masking - and the benefit was able to transfer when tested on a different corpus. We also found the masking preferably enhanced the accuracy for question-answer pairs of the top 20%-40% frequent in the train set. The results suggested that transformers-based QA systems may benefit from moderate masking during fine-tuning, likely by forcing the model to learn abstract context patterns rather than relying on specific surface terms or relations. The beneficial effect skewed toward a specific non-top frequency tier could reflect a more general phenomenon in machine learning where such enhancement techniques are most effective for cases that sit around the make-or-fail border.
Generic Action Start Detection

The online detection of action start in video data has witnessed an increase in attention from both academia and industry, for abundant use-cases (e.g., an alert mechanism in videos used for surveillance with an ability to automate the recording of key frames and timestamp). Conventional approaches heavily rely on frame-level annotations and other prior knowledge that can only be applied to limited categories. In this paper, we introduce Generic Action Start Detection (GASD): a new task that aims to detect the taxonomy-free action start in an online manner. Further-more, one novel yet simple design, 3D MLP-mixer based architecture with a multiscaled sampling training strategy, is proposed, which makes the GASD algorithm favorable for edge-device deployment. The GASD task is validated on two large-scale datasets, THUMOS'14 and ActivityNet1.2. Results demonstrate that the proposed architecture achieves the SOTA performance on the GASD task compared with other online action start detection algorithms.
Face security authentication based on specific expression action mode

Considering the growing security requirements, face security authentication (FSA) has attracted more and more researchers' attention, such as in financial payments, electronic devices unlock, secure access control, suspect matching and so on. At the same time, FSA is often seen as a kind of problem cost sensitive. This paper proposes a coarse-to-fine face security authentication method, which is named face authentication based on specific expression action mode (FSA_SEM) in cost sensitive learning. The highlights of this work are: (1) establish an identification dictionary to judge whether the visitor is member. (2) take a confirmation dictionary to confirm again whether the visitor is legitimate. (3) weight the dual dictionaries respectively by calculating the similarity relation matrix for adaptive weighting. The experimental results are encouraging, showing the effectiveness of FSA_SEM in different classification tasks. In practical application, the method can better deal with the cost sensitive FSA problem, and bring more convenient, safer and effective service for people's life.