Reviewerly/exHarmony huggingface dataset:
Systematic approach for thermal imaging camera calibration for machine vision applications

In order to estimate the intrinsic and extrinsic parameters of a camera, the geometric calibration method for a camera is required to map 3D world coordinate points onto 2D image coordinate points. In this work, a systematic approach for designing a calibration board is proposed to calibrate the thermal imaging cameras that include both mid-wave and long-wave infrared waveband (MWIR and LWIR). The low-cost materials such as thermocol, white glue, and aluminum foil sheet are used for our calibration board design and these materials have not been used for calibrating thermal cameras to the best of our knowledge. To achieve geometric calibration of thermal cameras, our calibration board has generated strong thermal image contrast and has displayed consistent checkerboard corner detection. The proposed method is tested in both indoor and outdoor environments with MWIR and LWIR cameras. The single camera calibrator tool and stereo camera calibrator tool in MATLAB are used for this work and it exhibited that the overall mean reprojection error was ≤ 0.40 pixel. The new board’s thermal imaging quality is compared to an existing paint-based calibration board. In terms of precision, low-cost construction and easy reusability, the proposed calibration board is proven to be more beneficial.
ETC-Oriented Efficient and Secure Blockchain: Credit-Based Mechanism and Evidence Framework for Vehicle Management

An electronic toll collection (ETC) system is one of the most important parts of an intelligent transportation system (ITS), but existing ETC systems are not efficient and have vehicle fee evasion complications. Owing to the traceability and tamper resistance of a blockchain, the combination of a blockchain and ETC system is a feasible way to solve the above problems. However, traditional blockchain (e.g., Bitcoin and Ethereum) has high power consumption and low efficiency. Hence, it cannot be used in ETC systems with high throughput and is not suitable for power-constrained Internet of Things (IoT) devices. In this study, therefore, we propose a blockchain architecture for the ETC system. To curb vehicle fee evasion behavior, thereby reducing the economic loss caused, we propose a vehicle behavior management mechanism based on credit value. Furthermore, to reduce the burden of storage and build a chain of evidence for auditing purposes, we propose an evidence chain framework. To protect the data security in the transaction process, we designed a data protection method to encrypt the transaction data. Additionally, this system is based on an open-source alliance blockchain framework called Hyperledger Fabric, which is more in line with the current application scenarios of ETC systems. We implemented the system on a Raspberry Pi and carried out simulations. The comprehensive evaluation results and analysis show that the system effectively reduces the number of illegal acts, completes the evidence inspection, and improves the security of the data.
Generating Synthetic Handwritten Historical Documents with OCR Constrained GANs

We present a framework to generate synthetic historical documents with precise ground truth using nothing more than a collection of unlabeled historical images. Obtaining large labeled datasets is often the limiting factor to effectively use supervised deep learning methods for Document Image Analysis (DIA). Prior approaches towards synthetic data generation either require human expertise or result in poor accuracy in the synthetic documents. To achieve high precision transformations without requiring expertise, we tackle the problem in two steps. First, we create template documents with user-specified content and structure. Second, we transfer the style of a collection of unlabeled historical images to these template documents while preserving their text and layout. We evaluate the use of our synthetic historical documents in a pre-training setting and find that we outperform the baselines (randomly initialized and pre-trained). Additionally, with visual examples, we demonstrate a high-quality synthesis that makes it possible to generate large labeled historical document datasets with precise ground truth.
Nuclei Grading of Clear Cell Renal Cell Carcinoma in Histopathological Image by Composite High-Resolution Network

The grade of clear cell renal cell carcinoma (ccRCC) is a critical prognostic factor, making ccRCC nuclei grading a crucial task in RCC pathology analysis. Computer-aided nuclei grading aims to improve pathologists’ work efficiency while reducing their misdiagnosis rate by automatically identifying the grades of tumor nuclei within histopathological images. Such a task requires precisely segment and accurately classify the nuclei. However, most of the existing nuclei segmentation and classification methods can not handle the inter-class similarity property of nuclei grading, thus can not be directly applied to the ccRCC grading task. In this paper, we propose a Composite High-Resolution Network for ccRCC nuclei grading. Specifically, we propose a segmentation network called W-Net that can separate the clustered nuclei. Then, we recast the fine-grained nuclei classification into two cross-category classification tasks that are leaned by two newly designed high-resolution feature extractors (HRFEs). The two HRFEs share the same backbone encoder with W-Net by a composite connection so that meaningful features for the segmentation task can be inherited to the classification task. Last, a head-fusion block is applied to generate the predicted label of each nucleus. Furthermore, we introduce a dataset for ccRCC nuclei grading, containing 1000 image patches with 70945 annotated nuclei. We demonstrate that our proposed method achieves state-of-the-art performance compared to existing methods on this large ccRCC grading dataset.
Hierarchical Self-supervised Learning for Medical Image Segmentation Based on Multi-domain Data Aggregation

A large labeled dataset is a key to the success of supervised deep learning, but for medical image segmentation, it is highly challenging to obtain sufficient annotated images for model training. In many scenarios, unannotated images are abundant and easy to acquire. Self-supervised learning (SSL) has shown great potentials in exploiting raw data information and representation learning. In this paper, we propose Hierarchical Self-Supervised Learning (HSSL), a new self-supervised framework that boosts medical image segmentation by making good use of unannotated data. Unlike the current literature on task-specific self-supervised pretraining followed by supervised fine-tuning, we utilize SSL to learn task-agnostic knowledge from heterogeneous data for various medical image segmentation tasks. Specifically, we first aggregate a dataset from several medical challenges, then pre-train the network in a self-supervised manner, and finally fine-tune on labeled data. We develop a new loss function by combining contrastive loss and classification loss, and pre-train an encoder-decoder architecture for segmentation tasks. Our extensive experiments show that multi-domain joint pre-training benefits downstream segmentation tasks and outperforms single-domain pre-training significantly. Compared to learning from scratch, our method yields better performance on various tasks (e.g., \(+0.69\%\) to \(+18.60\%\) in Dice with \(5\%\) of annotated data). With limited amounts of training data, our method can substantially bridge the performance gap with respect to denser annotations (e.g., \(10\%\) vs. \(100\%\) annotations).
A Chatbot to promote Students Mental Health through Emotion Recognition

The objective of this paper is to develop a chatbot for students to promote their mental health through emotion recognition technique. Nowadays, students are facing a lot of mental health issues due to various reasons like pandemic lockdowns, peer pressure, social media bullying, academic stress, loneliness, sexual harassment, etc. Because of which students are unable to progress well in their life, both emotionally and academically. Due to the battle of life, the students are also unable to receive proper guidance from experienced and knowledgeable humans to solve their personal issues. Therefore, developing a human friendly chatbot that can help students get the right guidance for their issues at the right time is a much needed one. These kind of chatbots can play a vital role in reducing the number of suicides in the country due to depression and stress. The chatbot named Maxx helps students solve or prevent any mental health issue in their day-to-day life. Maxx converses with the student, understands his/her present emotional state/mental health issue (if any), identifies the cause of that emotion/mental health issue and provides the right guidance based on the reason identified. Maxx uses technologies like DialogFlow for Natural Language Processing (NLP), Flutter for app development and Google Cloud Platform (GCP) for data storage and security.
A Confidence-based Iterative Solver of Depths and Surface Normals for Deep Multi-view Stereo

In this paper, we introduce a deep multi-view stereo (MVS) system that jointly predicts depths, surface normals and per-view confidence maps. The key to our approach is a novel solver that iteratively solves for per-view depth map and normal map by optimizing an energy potential based on the locally planar assumption. Specifically, the algorithm updates depth map by propagating from neigh-boring pixels with slanted planes, and updates normal map with local probabilistic plane fitting. Both two steps are monitored by a customized confidence map. This solver is not only effective as a post-processing tool for plane-based depth refinement and completion, but also differentiable such that it can be efficiently integrated into deep learning pipelines. Our multi-view stereo system employs multiple optimization steps of the solver over the initial prediction of depths and surface normals. The whole system can be trained end-to-end, decoupling the challenging problem of matching pixels within poorly textured regions from the cost-volume based neural network. Experimental results on ScanNet and RGB-D Scenes V2 demonstrate state-of-the-art performance of the proposed deep MVS system on multi-view depth estimation, with our proposed solver consistently improving the depth quality over both conventional and deep learning based MVS pipelines. Code is available at https://github.com/thuzhaowang/idn-solver.
Exploring and interrogating astrophysical data in virtual reality

Scientists across all disciplines increasingly rely on machine learning algorithms to analyse and sort datasets of ever increasing volume and complexity. Although trends and outliers are easily extracted, careful and close inspection will still be necessary to explore and disentangle detailed behaviour, as well as identify systematics and false positives. We must therefore incorporate new technologies to facilitate scientific analysis and exploration. Astrophysical data is inherently multi-parameter, with the spatial-kinematic dimensions at the core of observations and simulations. The arrival of mainstream virtual-reality (VR) headsets and increased GPU power, as well as the availability of versatile development tools for video games, has enabled scientists to deploy such technology to effectively interrogate and interact with complex data. In this paper we present development and results from custom-built interactive VR tools, called the iDaVIE suite, that are informed and driven by research on galaxy evolution, cosmic large-scale structure, galaxy–galaxy interactions, and gas/kinematics of nearby galaxies in survey and targeted observations. In the new era of Big Data ushered in by major facilities such as the SKA and LSST that render past analysis and refinement methods highly constrained, we believe that a paradigm shift to new software, technology and methods that exploit the power of visual perception, will play an increasingly important role in bridging the gap between statistical metrics and new discovery. We have released a beta version of the iDaVIE software system that is free and open to the community.
Visual Arrangements of Bar Charts Influence Comparisons in Viewer Takeaways

Well-designed data visualizations can lead to more powerful and intuitive processing by a viewer. To help a viewer intuitively compare values to quickly generate key takeaways, visualization designers can manipulate how data values are arranged in a chart to afford particular comparisons. Using simple bar charts as a case study, we empirically tested the comparison affordances of four common arrangements: vertically juxtaposed, horizontally juxtaposed, overlaid, and stacked. We asked participants to type out what patterns they perceived in a chart and we coded their takeaways into types of comparisons. In a second study, we asked data visualization design experts to predict which arrangement they would use to afford each type of comparison and found both alignments and mismatches with our findings. These results provide concrete guidelines for how both human designers and automatic chart recommendation systems can make visualizations that help viewers extract the "right" takeaway.
Revisiting image captioning via maximum discrepancy competition

Image captioning is a hot research topic bridging computer vision and natural language processing during the past several decades. It has achieved great progress with the help of large-scale datasets and deep learning techniques. Though the variety of image captioning models (ICMs), the performance of ICMs have got stuck in a bottleneck judging from the publicly published results. Considering the marginal performance gains brought by recent ICMs, we raise the following question: "what about the performances of the recent ICMs achieve on in-the-wild images? To clarify this question, we compare existing ICMs by evaluating their generalization ability. Specifically, we propose a novel method based on maximum discrepancy competition to diagnose existing ICMs. Firstly, we establish a new test set containing only informative images selected by adopting maximum discrepancy competition on the existing ICMs, from an arbitrary large-scale raw image set. Secondly, a small-scale and low-cost subjective annotation experiment is conducted on the new test set. Thirdly, we rank the generalization ability of the existing ICMs by comparing their performances on the new test set. Finally, the keys of different ICMs are demonstrated based on a detailed analysis of experimental results. Our analysis yields several interesting findings, including that 1) Using simultaneously low- and high-level object features may be an effective tool to boost the generalization ability for the Transformer based ICMs. 2) Self-attention mechanism may provide better modelling ability for inter- and intra-modal data than other attention-based mechanisms. 3) Constructing an ICM with a multistage language decoder may be a promising way to improve its performance.
Artificial Intelligence, Social Media and Supply Chain Management: The Way Forward

Supply chain management (SCM) is a complex network of multiple entities ranging from business partners to end consumers. These stakeholders frequently use social media platforms, such as Twitter and Facebook, to voice their opinions and concerns. AI-based applications, such as sentiment analysis, allow us to extract relevant information from these deliberations. We argue that the context-specific application of AI, compared to generic approaches, is more efficient in retrieving meaningful insights from social media data for SCM. We present a conceptual overview of prevalent techniques and available resources for information extraction. Subsequently, we have identified specific areas of SCM where context-aware sentiment analysis can enhance the overall efficiency.
Learning Multiple Pixelwise Tasks Based on Loss Scale Balancing

We propose a novel loss weighting algorithm, called loss scale balancing (LSB), for multi-task learning (MTL) of pixelwise vision tasks. An MTL model is trained to estimate multiple pixelwise predictions using an overall loss, which is a linear combination of individual task losses. The proposed algorithm dynamically adjusts the linear weights to learn all tasks effectively. Instead of controlling the trend of each loss value directly, we balance the loss scale — the product of the loss value and its weight — periodically. In addition, by evaluating the difficulty of each task based on the previous loss record, the proposed algorithm focuses more on difficult tasks during training. Experimental results show that the proposed algorithm outperforms conventional weighting algorithms for MTL of various pixelwise tasks. Codes are available at https://github.com/jaehanlee-mcl/LSB-MTL.
Detecting Verbal and Non-Verbal Gestures Using Earables

Verbal and non-verbal activities convey insightful information about people's affect, empathy, and engagement during social interactions. In this paper, we investigate the usage of inertial sensors to recognize verbal (e.g., speaking), non-verbal (e.g., head nodding, shaking) and other activities (e.g., eating, no movement). We implement an end-to-end deep neural network to distinguish among these activities. We then explore the generalizability of the approach in three scenarios: (1) using new data to detect a known activity from a known user, (2) detecting a novel activity of a known user and (3) detecting the activity of an unknown user. Results show that using accelerometer and gyroscope sensors, the model achieves a balanced accuracy of 55% when tested on data from a new user, 41% on a new activity of an existing user, and 80% on new data of a known activity from an existing user. The results are between 7-47 percentage points higher than baseline classifiers.
E3XR: An Analytical Framework for Ethical, Educational and Eudaimonic XR Design

A rapidly growing number of educators and students now embrace XR as a powerful technology with affordances that can support many benefits, including highly immersive learning experiences, empathy and perspectives on social issues; XR can be designed in ways that can provide new pathways to success and opportunity. Yet the mirror image is also true -- XR can be designed in ways that lead to increased risk, perpetuation of inequities and other harmful impacts to individuals and society. We need ways to analyze XR in terms of ethical aspects, educational efficacy and whether it supports or hinders human flourishing (i.e., eudaimonia). In this paper, we discuss XR as a double-edged sword that can be leveraged for positive or negative outcomes, whether intentionally or unintentionally; that is, we highlight various opportunities and benefits at hand, but also risks and possible negative impacts. We introduce E3XR, a framework that serves as an analytical lens to determine the ethics, learning theory and human flourishing aspects of an XR design. For each component of this framework, we review relevant literature and consider the threats and opportunities that can be evaluated. Finally, we conclude with a discussion of the significance of this work and implications for designers and educators.
Learning Quick Fixes from Code Repositories

Code analyzers such as Error Prone and FindBugs detect code patterns symptomatic of bugs, performance issues, or bad style. These tools express patterns as quick fixes that detect and rewrite unwanted code. However, it is difficult to come up with new quick fixes and decide which ones are useful and frequently appear in real code. We propose to rely on the collective wisdom of programmers and learn quick fixes from revision histories in software repositories. We present Revisar, a tool for discovering common Java edit patterns in code repositories. Given code repositories and their revision histories, Revisar (i) identifies code edits from revisions and (ii) clusters edits into sets that can be described using an edit pattern. The designers of code analyzers can then inspect the patterns and add the corresponding quick fixes to their tools. We ran Revisar on nine popular GitHub projects, and it discovered 89 useful edit patterns that appeared in 3 or more projects. Moreover, 64% of the discovered patterns did not appear in existing tools. We then conducted a survey with 164 programmers from 124 projects and found that programmers significantly preferred eight out of the nine of the discovered patterns. Finally, we submitted 16 pull requests applying our patterns to 9 projects and, at the time of the writing, programmers accepted 7 (63.6%) of them. The results of this work aid toolsmiths in discovering quick fixes and making informed decisions about which quick fixes to prioritize based on patterns programmers actually apply in practice.
Next-Generation Data Center Network Enabled by Machine Learning: Review, Challenges, and Opportunities

Data center network (DCN) is the backbone of many emerging applications from smart connected homes to smart traffic control and is continuously evolving to meet the diverse and ever-increasing computing requirements of these applications. The data centers often have tens of thousands of components such as servers and switches/routers that work together to achieve a common objective and serve these applications. Managing such large data centers is a tedious process and demands for automation, intelligent control and decision making within the data center. Recently both the industry and academia have focused on bringing intelligence to the control, automation and management of DCNs. Despite the variety of works that surveyed ML for networking, to the best of our knowledge, none has focused on DCN, which makes this survey original. Readers in the academic and industrial communities will all benefit from a comprehensive discussion of the ML solutions applied in DCN to address critical essential problems, including workload forecasting, traffic control and optimization, topology management, network state prediction and failures analysis, and security. Furthermore, this article outlines the challenges and concludes with the future research venues in adopting ML for automatic, intelligent and autonomous DCNs.
Joint <i>t</i>-SNE for Comparable Projections of Multiple High-Dimensional Datasets

We present Joint t-Stochastic Neighbor Embedding (Joint t-SNE), a technique to generate comparable projections of multiple high-dimensional datasets. Although t-SNE has been widely employed to visualize high-dimensional datasets from various domains, it is limited to projecting a single dataset. When a series of high-dimensional datasets, such as datasets changing over time, is projected independently using t-SNE, misaligned layouts are obtained. Even items with identical features across datasets are projected to different locations, making the technique unsuitable for comparison tasks. To tackle this problem, we introduce edge similarity, which captures the similarities between two adjacent time frames based on the Graphlet Frequency Distribution (GFD). We then integrate a novel loss term into the t-SNE loss function, which we call vector constraints, to preserve the vectors between projected points across the projections, allowing these points to serve as visual landmarks for direct comparisons between projections. Using synthetic datasets whose ground-truth structures are known, we show that Joint t-SNE outperforms existing techniques, including Dynamic t-SNE, in terms of local coherence error, Kullback-Leibler divergence, and neighborhood preservation. We also showcase a real-world use case to visualize and compare the activation of different layers of a neural network.