Reviewerly/exHarmony huggingface dataset:
DesmokeNet: A Two-Stage Smoke Removal Pipeline Based on Self-Attentive Feature Consensus and Multi-Level Contrastive Regularization

In image processing, smoke may degrade visibility and deteriorate the performance of high-level vision applications. Therefore, single image smoke removal is crucial for computer vision. Currently, existing smoke removal algorithms mainly leverage handcrafted priors. Moreover, these methods usually apply haze removal methods to perform smoke removal due to the similarity between smoke and haze. However, these methods cannot sufficiently address the degradation of thick smoke and may suffer from residual smoke and color distortion problems due to the non-global and non-homogeneous distribution of smoke. In this paper, to solve the aforementioned problems, an end-to-end deep neural network called DesmokeNet is proposed. We construct a two-stage recovered pipeline to remove the smoke in different thicknesses. The light and thick smoke is first removed locally by the smoke removal network (SRN). The missing pixels in the thick smoke are then recovered by the pixel compensation network (PCN). Moreover, we proposed the thickness-aware pixel loss and the dark channel loss to suppress the residual smoke. To further increase the discriminative ability of the DesmokeNet, we proposed self-attentive feature consensus loss and multi-level contrastive regularization loss to improve the performance of smoke removal. Finally, to train the proposed method, we construct the first large-scale dataset containing synthetic and real-world data. Extensive experiments show that the proposed method outperforms favorably against other state-of-the-art methods quantitatively and qualitatively.
An efficient image super resolution model with dense skip connections between complex filter structures in Generative Adversarial Networks

Generative Adversarial Network (GAN) based models have gained a lot of popularity due to their outstanding performance in image super resolution tasks. However, these networks have few inherent problems; such as, high computational complexity, large depth and vanishing gradient. Present work proposes a novel GAN based architecture, namely, Super Resolution with Inception Network (SRINet) to solve the above mentioned problems. Generator architecture of SRINet uses complex filter structure rather than the linear filter structure to increase the depth and width of network without increasing the computational cost. Complex filter settings, present in the architecture, helps it to attain locally distributed information along with hierarchical global information in an image. Hence, the proposed method approximates the most favorable sparse structures to foster the learning capability of network. Additionally, progressive two stage upscaling approach with dense skip connections are introduced in the generator architecture. This technique helps the proposed network to learn precise mapping to generate an output image from the low resolution image. To measure visual quality of an image, we use human visual system based visual information fidelity metric. Proposed method outperforms all the state-of-the-art methods qualitatively (perceptually) and quantitatively on other GAN based methods.
DeLFIQE—A Low-Complexity Deep Learning-Based Light Field Image Quality Evaluator

Light field image quality assessment (LF-IQA) has attracted increasing research interests due to the fast-growing demands for immersive media experience. The existing LF-IQA metrics, however, heavily rely on high-complexity statistics-based feature extraction, which is computationally intensive and hardly sustainable in real-time systems. In this article, we propose a low-complexity deep learning-based light field image quality evaluator (DeLFIQE) that achieves both improved accuracy and significantly reduced complexity in measuring LF image quality. On one hand, we propose the novel discriminative epipolar plane image (D-EPI) patches, which reduces the high-complexity 4-D light field IQA processing to lower dimension and lower complexity 2-D IQA processing. On the other hand, we design a distortion-aware multitask learning scheme in DeLFIQE to improve the quality measurement accuracy and generalization. Specifically, alongside the main task that predicts the light field image (LFI) quality, we design an auxiliary classification task that classifies the D-EPI patches based on their distortion types and severity levels, which we believe both are key factors affecting the LFI quality. To the best of our knowledge, this is the first work in LF-IQA with a dedicatedly designed deep multitask convolutional neural network (CNN) model. Our extensive experimental results show that DeLFIQE outperforms the state-of-the-art metrics on representative benchmark datasets with significantly reduced complexity.
Development of the Advanced Encryption Standard

Strong cryptographic algorithms are essential for the protection of stored and transmitted data throughout the world. This publication discusses the development of Federal Information Processing Standards Publication (FIPS) 197, which specifies a cryptographic algorithm known as the Advanced Encryption Standard (AES). The AES was the result of a cooperative multiyear effort involving the U.S. government, industry, and the academic community. Several difficult problems that had to be resolved during the standard's development are discussed, and the eventual solutions are presented. The author writes from his viewpoint as former leader of the Security Technology Group and later as acting director of the Computer Security Division at the National Institute of Standards and Technology, where he was responsible for the AES development.
Deepfake Video Detection Using 3D-Attentional Inception Convolutional Neural Network

The current spike of deepfake techniques has received considerable attention due to security concerns. To mitigate the potential risks brought by deepfake techniques, many detection methods have been proposed. However, most existing works merely leverage spatial information from separate frames and ignore valuable inter-frame temporal information. In this paper, we propose a deepfake detection scheme that uses 3D-attentional inception network. The proposed model encompasses both spatial and temporal information simultaneously with the 3D kernels. Furthermore, the channel and spatial-temporal attention modules are applied to improve detection capabilities. Comprehensive experiments demonstrate that our scheme outperforms state-of-the-art methods.
Toward static test flakiness prediction: a feasibility study

Flaky tests are tests that exhibit both a passing and failing behavior when run against the same code. While the research community has attempted to define automated approaches for detecting and addressing test flakiness, most of them suffer from scalability issues and uncertainty as they require test cases to be run multiple times. This limitation has been recently targeted by means of machine learning solutions that could predict the flakiness of tests using a set of both static and dynamic metrics that would avoid the re-execution of tests. Recognizing the effort spent so far, this paper poses the first steps toward an orthogonal view of the problem, namely the classification of flaky tests using only statically computable software metrics. We propose a feasibility study on 72 projects of the iDFlakies dataset, and investigate the differences between flaky and non-flaky tests in terms of 25 test and production code metrics and smells. First, we statistically assess those differences. Second, we build a logistic regression model to verify the extent to which the differences observed are still significant when the metrics are considered together. The results show a relation between test flakiness and a number of test and production code factors, indicating the possibility to build classification approaches that exploit those factors to predict test flakiness.
Improving Filling Level Classification with Adversarial Training

We investigate the problem of classifying–from a single image–the level of content in a cup or a drinking glass. This problem is made challenging by several ambiguities caused by transparencies, shape variations and partial occlusions, and by the availability of only small training datasets. In this paper, we tackle this problem with an appropriate strategy for transfer learning. Specifically, we use adversarial training in a generic source dataset and then refine the training with a task-specific dataset. We also discuss and experimentally evaluate several training strategies and their combination on a range of container types of the CORSMAL Containers Manipulation dataset. We show that transfer learning with adversarial training in the source domain consistently improves the classification accuracy on the test set and limits the overfitting of the classifier to specific features of the training data.
GLIMG: Global and local item graphs for top-N recommender systems

Graph-based recommendation models work well for top-N recommender systems due to their capability to capture the potential relationships between entities. However, most of the existing methods only construct a single global item graph shared by all the users and regrettably ignore the diverse tastes between different user groups. Inspired by the success of local models for recommendation tasks, this paper provides the first attempt to investigate multiple local item graphs along with a global item graph for graph-based recommendation models. We argue that recommendation on global and local graphs outperforms that on a single global graph or multiple local graphs. Specifically, we propose a novel graph-based recommendation model named GLIMG (Global and Local IteM Graphs), which simultaneously captures both the global and local user tastes. By integrating the global and local graphs into an adapted semi-supervised learning model, users' preferences on items are propagated globally and locally. Extensive experimental results on real-world datasets show that our proposed method consistently outperforms the state-of-the-art counterparts on the top-N recommendation task.
Energy-Efficient Neural Network Inference with Microcavity Exciton Polaritons

We propose all-optical neural networks characterized by very high energy efficiency and performance density of inference. We argue that the use of microcavity exciton-polaritons allows to take advantage of the properties of both photons and electrons in a seamless manner. This results in strong optical nonlinearity without the use of optoelectronic conversion. We propose a design of a realistic neural network and estimate energy cost to be at the level of attojoules per bit, also when including the optoelectronic conversion at the input and output of the network, several orders of magnitude below state-of-the-art hardware implementations. We propose two kinds of nonlinear binarized nodes based either on optical phase shifts and interferometry or on polariton spin rotations.
To determine the critical factors for the adoption of cloud computing in the educational sector in developing countries – a fuzzy DEMATEL approach

Purpose The paper focuses on reviewing and theorizing the factors that affect the adoption of cloud computing in the education sector narrowing the focus to developing countries such as India. Design/methodology/approach Through an extensive literature survey, critical factors of cloud computing for education were identified. Further, the fuzzy DEMATEL approach was used to define their interrelationship and its cause and effect. Findings A total of 17 factors were identified for the study based on the literature survey and experts' input. These factors were classified as causes and effects and ranked and interrelated. “Required Learning Skills and Attitude,” “Lack of Infrastructure,” “Learners' Ability” and “Increased Investment” are found to be the most influential factors. Practical implications The resultant ranking factors can be used as a basis for managing the process of cloud adoption in several institutions. The study could guide academicians, policymakers and government authorities for the effective adoption of cloud computing in education. Originality/value The study investigates interdependency amongst the factors of cloud computing for education in context with developing economy. This is one of first study in higher education institutes of India.
Cross-Modal Transmission Strategy

Multi-modal services, typically integrated by visual, audio, and haptic signals, have been considered as promising killer services in 5G and beyond 5G era. However, due to essential difference among these signals, how to guarantee quality of multi-modal services is a significant technical challenge. Existing transmission schemes, which deliver and process each modality signal separately, cannot meet such requirements as low latency, high reliability, and high throughput. To get over the dilemma, this paper proposes a general cross-modal transmission strategy by taking advantage of the potential correlation among modalities, which consists of a delivery mechanism at the sender and a signal restoration procedure at the receiver. On the one hand, a visual-aided haptic content compression method is designed for the delivery mechanism. By utilizing the category correlation among modalities, haptic signals with similar visual content can be effectively compressed, reducing transmission burden. On the other hand, a fine-grained haptic to image synthesis (FHIS) approach is proposed for realizing signal restoration. Through exploring strong matching properties among modalities, the FHIS can restore the impaired, missing, delayed visual images from the received haptic signals. Experiments on a standard visual-haptic database and a practical platform evaluate the performance of the constructed cross-modal transmission strategy.
Guided pattern mining for API misuse detection by change-based code analysis

Abstract Lack of experience, inadequate documentation, and sub-optimal API design frequently cause developers to make mistakes when re-using third-party implementations. Such API misuses can result in unintended behavior, performance losses, or software crashes. Therefore, current research aims to automatically detect such misuses by comparing the way a developer used an API to previously inferred patterns of the correct API usage. While research has made significant progress, these techniques have not yet been adopted in practice. In part, this is due to the lack of a process capable of seamlessly integrating with software development processes. Particularly, existing approaches do not consider how to collect relevant source code samples from which to infer patterns. In fact, an inadequate collection can cause API usage pattern miners to infer irrelevant patterns which leads to false alarms instead of finding true API misuses. In this paper, we target this problem (a) by providing a method that increases the likelihood of finding relevant and true-positive patterns concerning a given set of code changes and agnostic to a concrete static, intra-procedural mining technique and (b) by introducing a concept for just-in-time API misuse detection which analyzes changes at the time of commit. Particularly, we introduce different, lightweight code search and filtering strategies and evaluate them on two real-world API misuse datasets to determine their usefulness in finding relevant intra-procedural API usage patterns. Our main results are (1) commit-based search with subsequent filtering effectively decreases the amount of code to be analyzed, (2) in particular method-level filtering is superior to file-level filtering, (3) project-internal and project-external code search find solutions for different types of misuses and thus are complementary, (4) incorporating prior knowledge of the misused API into the search has a negligible effect.
Deep momentum uncertainty hashing

Combinatorial optimization (CO) has been a hot research topic because of its theoretic and practical importance. As a classic CO problem, deep hashing aims to find an optimal code for each data from finite discrete possibilities, while the discrete nature brings a big challenge to the optimization process. Previous methods usually mitigate this challenge by binary approximation, substituting binary codes for real-values via activation functions or regularizations. However, such approximation leads to uncertainty between real-values and binary ones, degrading retrieval performance. In this paper, we propose a novel Deep Momentum Uncertainty Hashing (DMUH). It explicitly estimates the uncertainty during training and leverages the uncertainty information to guide the approximation process. Specifically, we model bit-level uncertainty via measuring the discrepancy between the output of a hashing network and that of a momentum-updated network. The discrepancy of each bit indicates the uncertainty of the hashing network to the approximate output of that bit. Meanwhile, the mean discrepancy of all bits in a hashing code can be regarded as image-level uncertainty. It embodies the uncertainty of the hashing network to the corresponding input image. The hashing bit and image with higher uncertainty are paid more attention during optimization. To the best of our knowledge, this is the first work to study the uncertainty in hashing bits. Extensive experiments are conducted on four datasets to verify the superiority of our method, including CIFAR-10, NUS-WIDE, MS-COCO, and a million-scale dataset Clothing1M. Our method achieves the best performance on all of the datasets and surpasses existing state-of-the-art methods by a large margin.
BCA: A 530-mW Multicore Blockchain Accelerator for Power-Constrained Devices in Securing Decentralized Networks

Blockchain distributed ledger technology (DLT) has widespread applications in society 5.0 because it improves service efficiency and significantly reduces labor costs. However, employing blockchain DLT entails considerable energy consumption in the mining process. This paper proposes a blockchain accelerator (BCA) with ultralow power consumption and a high processing rate to address the problem. The BCA focuses on accelerating the double secure hash algorithm (SHA) 256 function required in the mining process at a system-on-chip (SoC) level. We propose three ideas, namely, multiple local memories (multimem), double-cell processing element (D-PE), and nonce autoupdate (NAU), to reduce the external data transfer time and improve the BCA hardware efficiency. We propose a cascaded multiple BCA chip model to enhance the system throughput by several-fold. Our experiments on an ASIC and FPGA prove that the proposed BCA successfully performs the mining process for multiple blockchain networks with much lower power consumption than that of the state-of-the-art CPUs and GPUs. The BCA is laid out with Renesas 65 nm technology with a chip area of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$25~mm^{2}$ </tex-math></inline-formula> and consumes <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$530~mW$ </tex-math></inline-formula> at 100 <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MHz</i> . The power efficiency of the layout chip is improved by 2428 and 143 times compared with that of the fastest CPU Intel i9-10940X and GPU RTX 3090, respectively.
Multimodal Virtual Avatars for Investigative Interviews with Children

In this article, we present our ongoing work in the field of training police officers who conduct interviews with abused children. The objectives in this context are to protect vulnerable children from abuse, facilitate prosecution of offenders, and ensure that innocent adults are not accused of criminal acts. There is therefore a need for more data that can be used for improved interviewer training to equip police with the skills to conduct high-quality interviews. To support this important task, we propose to research a training program that utilizes different system components and multimodal data from the field of artificial intelligence such as chatbots, generation of visual content, text-to-speech, and speech-to-text. This program will be able to generate an almost unlimited amount of interview and also training data. The goal of combining all these different technologies and datatypes is to create an immersive and interactive child avatar that responds in a realistic way, to help to support the training of police interviewers, but can also produce synthetic data of interview situations that can be used to solve different problems in the same domain.
Suppressing Biased Samples for Robust VQA

Most existing visual question answering (VQA) models strongly rely on language bias to answer questions, i.e., they always tend to fit question-answer pairs on the train split and perform poorly on the test spilt when the answer distributions are different. This behavior makes them hard to be applied in real scenarios. To reduce the language biases, previous studies mainly integrate modules to overcome language priors (ensemble-based methods) or generate additional training data to balance dataset biases (data-balanced methods). However, all the existing ensemble-based methods drop their accuracies on the VQA v2 dataset, while data-balanced methods may introduce new biases and cannot guarantee the quality of the generated data. In this paper, we propose a model-agnostic training scheme called Suppressing Biased Samples (SBS) to overcome language priors. SBS consists of two collaborative parts, i.e., a Data Classifier Module to divide the dataset into biased samples and unbiased samples by utilizing the similarity in the semantic space, and a Bias Penalty Module to suppress the biased samples to weaken their influence. As a new way of balancing data to address language bias, SBS overcomes the shortcomings of previous data-balanced methods. Experimental results show that our method can be merged into other bias-reduction methods and achieves a new state-of-the-art performance on the commonly used VQA-CP v2 dataset.
LifeConcept: An Interactive Approach for Multimodal Lifelog Retrieval through Concept Recommendation

The major challenge in visual lifelog retrieval is the semantic gap between textual queries and visual concepts. This paper presents our work on the Lifelog Search Challenge 2021 (LSC'21), an annual comparative benchmarking activity for comparing approaches to interactive retrieval from multimodal lifelogs. We propose LifeConcept, an interactive lifelog search system that is aimed at accelerating the retrieval process and retrieving more precise results. In this work, we introduce several new features such as the number of people, location cluster, and object with color. Moreover, we obtain visual concepts from the images with computer vision models and propose a concept recommendation method to reduce the semantic gap. In this way, users can efficiently set up the related conditions for their requirements and search the desired images with appropriate query terms based on the suggestion.