Reviewerly/exHarmony huggingface dataset:
FPGA-based Acceleration for Convolutional Neural Networks on PYNQ-Z2

Convolutional neural network is now widely used in computer vision and deep learning applications.The most computeintensive layer in convolutional neural networks is the convolutional layer, which should be accelerated in hardware.This paper aims to develop an efficient hardware-software co-design framework for machine learning applications on the PYNQ-Z2 board.To achieve this goal, we develop hardware implementations of convolutional IP core and use them as Python overlays.Experiments show that the hardware implementations of the convolutional IP core outperform their software implementations by factors of up to 9 times.Furthermore, we make use of the designed convolutional IP core as hardware accelerator in the handwritten digit recognition application with MNIST dataset.Thanks to the use of the hardware accelerator for the convolutional layers, the execution performance of the convolutional neural network has been improved by a factor of 6.2 times.
Comparative analysis of contextual and context-free embeddings in disaster prediction from Twitter data

Twitter is a social media site where people post their personal experiences, opinions, and news. Due to the ubiquitous real-time data availability, many rescue agencies monitor this data regularly to identify disasters, reduce risk, and save lives. However, it is impossible for humans to manually check the mass amount of data and identify disasters in real-time. For this purpose, many research have been proposed to present words in machine-understandable representations and apply machine learning methods on the word representations to identify the sentiment of a text. The previous research methods provide a single vector representation or embedding of a word from a given document. However, the recent advanced contextual embedding method (BERT — Bidirectional Encoder Representations from Transformers) constructs different vectors for the same word in different contexts. The BERT embeddings have been used successfully in various Natural Language Processing (NLP) tasks, yet there is no concrete analysis of how these representations are helpful in disaster-type tweet analysis. This research study explores the efficacy of the BERT embeddings on predicting disaster from Twitter data and compares these to traditional context-free word embedding methods. We provide both quantitative and qualitative results for this study. The results show that the contextual embeddings have the best results in disaster prediction task than the traditional word embeddings. Furthermore, we discuss the opportunities and challenges of contextual embeddings on sentiment analysis of Twitter data.
A Free Format Legal Question Answering System

We present an information retrieval-based question answer system to answer legal questions.The system is not limited to a predefined set of questions or patterns and uses both sparse vector search and embeddings for input to a BERT-based answer re-ranking system.A combination of general domain and legal domain data is used for training.This natural question answering system is in production and is used commercially.
Dynamic Invariant-Specific Representation Fusion Network for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) aims to infer emotions from linguistic, auditory, and visual sequences. Multimodal information representation method and fusion technology are keys to MSA. However, the problem of difficulty in fully obtaining heterogeneous data interactions in MSA usually exists. To solve these problems, a new framework, namely, dynamic invariant-specific representation fusion network (DISRFN), is put forward in this study. Firstly, in order to effectively utilize redundant information, the joint domain separation representations of all modes are obtained through the improved joint domain separation network. Then, the hierarchical graph fusion net (HGFN) is used for dynamically fusing each representation to obtain the interaction of multimodal data for guidance in the sentiment analysis. Moreover, comparative experiments are performed on popular MSA data sets MOSI and MOSEI, and the research on fusion strategy, loss function ablation, and similarity loss function analysis experiments is designed. The experimental results verify the effectiveness of the DISRFN framework and loss function.
Designing a clinical decision support system for Alzheimer’s diagnosis on OASIS-3 data set

Background and Objective: Alzheimer's disease (AD) is the most common neurodegenerative disease, and its early detection is crucial for appropriate treatment. To analyse 3D-magnetic resonance imaging (MRI) data, deep learning (DL) methods have become powerful tools. In this paper, we propose a clinical decision support system (CDSS) based on DL methods for diagnosing AD using 3D-MRI images. Methods: We conducted several experiments to choose the best model to use as the backbone of our CDSS. For this purpose, we utilize existing convolutional neural network (CNN), ResNet, DenseNet, and Inception-v3 models. We evaluated the models on the recently released part 3 of the Open Access Series of Imaging Studies (OASIS-3) data set. On analysis of the results, we propose a novel network which performed better than the tested models. Results: Compared to the mentioned networks, the proposed model performed best in AD classification. The 3D image inputs alongside clinical factors in our proposed ensemble architecture increased the performance of the model. The trained version of the proposed model with a graphical interface is proposed as a CDSS to help physicians. We also used person disjoint subsets of the data to avoid reporting biased performance of our research work. Conclusions: Our results show that our proposed model significantly enhances the precision of clinical examinations and makes the process more robust. This CDSS can potentially help to identify AD subjects with high confidence. To the best of our knowledge, this is the first comprehensive work on OASIS-3 data set, with significant results.
TIM: threat context-enhanced TTP intelligence mining on unstructured threat data

Abstract TTPs (Tactics, Techniques, and Procedures), which represent an attacker’s goals and methods, are the long period and essential feature of the attacker. Defenders can use TTP intelligence to perform the penetration test and compensate for defense deficiency. However, most TTP intelligence is described in unstructured threat data, such as APT analysis reports. Manually converting natural language TTPs descriptions to standard TTP names, such as ATT&amp;CK TTP names and IDs, is time-consuming and requires deep expertise. In this paper, we define the TTP classification task as a sentence classification task. We annotate a new sentence-level TTP dataset with 6 categories and 6061 TTP descriptions from 10761 security analysis reports. We construct a threat context-enhanced TTP intelligence mining (TIM) framework to mine TTP intelligence from unstructured threat data. The TIM framework uses TCENet (Threat Context Enhanced Network) to find and classify TTP descriptions, which we define as three continuous sentences, from textual data. Meanwhile, we use the element features of TTP in the descriptions to enhance the TTPs classification accuracy of TCENet. The evaluation result shows that the average classification accuracy of our proposed method on the 6 TTP categories reaches 0.941 . The evaluation results also show that adding TTP element features can improve our classification accuracy compared to using only text features. TCENet also achieved the best results compared to the previous document-level TTP classification works and other popular text classification methods, even in the case of few-shot training samples. Finally, the TIM framework organizes TTP descriptions and TTP elements into STIX 2.1 format as final TTP intelligence for sharing the long-period and essential attack behavior characteristics of attackers. In addition, we transform TTP intelligence into sigma detection rules for attack behavior detection. Such TTP intelligence and rules can help defenders deploy long-term effective threat detection and perform more realistic attack simulations to strengthen defense.
An Introduction to Digital Face Manipulation

Abstract Digital manipulation has become a thriving topic in the last few years, especially after the popularity of the term DeepFakes. This chapter introduces the prominent digital manipulations with special emphasis on the facial content due to their large number of possible applications. Specifically, we cover the principles of six types of digital face manipulations: (i) entire face synthesis, (ii) identity swap, (iii) face morphing, (iv) attribute manipulation, (v) expression swap (a.k.a. face reenactment or talking faces), and (vi) audio- and text-to-video. These six main types of face manipulation are well established by the research community, having received the most attention in the last few years. In addition, we highlight in this chapter publicly available databases and code for the generation of digital fake content.
Sentiment Analysis and Topic Modeling on Arabic Twitter Data during Covid-19 Pandemic

Twitter Sentiment Analysis is the task of detecting opinions and sentiments in tweets using different algorithms. In our research work, we conducted a study to analyze and compare different Algorithms of Machine Learning (MLAs) for the classification task, and hence we collected 37 875 Moroccan tweets, during the COVID-19 pandemic, from 01 March 2020 to 28 June 2020. The analysis was done using six classification algorithms (Naive Bayes, Logistic Regression, Support Vector Machine, K-Nearest Neighbors, Decision Tree, Random Forest classifier) and considering Accuracy, Recall, Precision, and F-Score as evaluation parameters. Then we applied topic modeling over the three classified tweets categories (negative, positive, and neutral) using Latent Dirichlet Allocation (LDA) which is among the most effective approaches to extract discussed topics. As result, the logistic regression classifier gave the best predictions of sentiments with an accuracy of 68.80%.
A novel deep ordinal classification approach for aesthetic quality control classification

Abstract Nowadays, decision support systems (DSSs) are widely used in several application domains, from industrial to healthcare and medicine fields. Concerning the industrial scenario, we propose a DSS oriented to the aesthetic quality control (AQC) task, which has quickly established itself as one of the most crucial challenges of Industry 4.0. Taking into account the increasing amount of data in this domain, the application of machine learning (ML) and deep learning (DL) techniques offers great opportunities to automatize the overall AQC process. State-of-the-art is mainly oriented to approach this problem with a nominal DL classification method which does not exploit the ordinal structure of the AQC task, thus not penalizing the error among distant AQC classes (which is a relevant aspect for the real use case). The paper introduces a DL ordinal methodology for the AQC classification. Differently from other deep ordinal methods, we combined the standard categorical cross-entropy with the cumulative link model and we imposed the ordinal constraint via the thresholds and slope parameters. Experimental results were performed for solving an AQC task on a novel image dataset originated from a specific company’s demand (i.e., aesthetic assessment of wooden stocks). We demonstrated how the proposed methodology is able to reduce misclassification errors (up to 0.937 quadratic weight kappa loss) among distant classes while overcoming other state-of-the-art deep ordinal models and reducing the bias factor related to the item geometry. The proposed DL approach was integrated as the main core of a DSS supported by Internet of Things (IoT) architecture that can support the human operator by reducing up to 90% the time needed for the qualitative analysis carried out manually in this specific domain.
Learning to reduce false positives in analytic bug detectors

Due to increasingly complex software design and rapid iterative development, code defects and security vulnerabilities are prevalent in modern software. In response, programmers rely on static analysis tools to regularly scan their codebases and find potential bugs. In order to maximize coverage, however, these tools generally tend to report a significant number of false positives, requiring developers to manually verify each warning. To address this problem, we propose a Transformer-based learning approach to identify false positive bug warnings. We demonstrate that our models can improve the precision of static analysis by 17.5%. In addition, we validated the generalizability of this approach across two major bug types: null dereference and resource leak.
Improved image classification explainability with high-accuracy heatmaps

Deep learning models have become increasingly used for image-based classification. In critical applications such as medical imaging, it is important to convey the reasoning behind the models' decisions in human-understandable forms. In this work, we propose Pyramid Localization Network (PYLON), a deep learning model that delivers precise location explanation by increasing the resolution of heatmaps produced by class activation map (CAM). PYLON substantially improves the quality of CAM's heatmaps in both general image and medical image domains and excels at pinpointing the locations of small objects. Most importantly, PYLON does not require expert annotation of the object location but instead can be trained using only image-level label. This capability is especially important for domain where expert annotation is often unavailable or costly to obtain. We also demonstrate an effective transfer learning approach for applying PYLON on small datasets and summarize technical guidelines that would facilitate wider adoption of the technique.
Neural Compression-Based Feature Learning for Video Restoration

How to efficiently utilize the temporal features is crucial, yet challenging, for video restoration. The temporal features usually contain various noisy and uncorrelated information, and they may interfere with the restoration of the current frame. This paper proposes learning noiserobust feature representations to help video restoration. We are inspired by that the neural codec is a natural denoiser: In neural codec, the noisy and uncorrelated contents which are hard to predict but cost lots of bits are more inclined to be discarded for bitrate saving. Therefore, we design a neural compression module to filter the noise and keep the most useful information in features for video restoration. To achieve robustness to noise, our compression module adopts a spatial-channel-wise quantization mechanism to adaptively determine the quantization step size for each position in the latent. Experiments show that our method can significantly boost the performance on video denoising, where we obtain 0.13 dB improvement over BasicVSR++ with only 0.23x FLOPs. Meanwhile, our method also obtains SOTA results on video deraining and dehazing.
Shared Virtual Reality Experiences during the COVID-19 Pandemic: Exploring the Gratifications and Effects of Engagement with Immersive Videos

The coronavirus (COVID-19) pandemic and recent economic recession have been impacting many people’s mental health. The experience of social distancing created new hardships for people who already reported symptoms of depression or anxiety. In these circumstances, new technologies, such as immersive virtual reality (VR) videos, could serve as useful tools for facilitating interactions, emotional sharing, and information processing within a virtual environment. In this study, researchers aimed to enrich the information processing literature by focusing on the uses and gratifications of 360-degree VR videos during the pandemic. Through employing survey research with 1422 participants located in the U.S. and structural equation modeling for data analysis, this study found that five types of gratification, including utilitarian (i.e., navigation), hedonic (i.e., enjoyment), sensual (i.e., realism), social (i.e., community), and symbolic (i.e., coolness), significantly motivated users to use such immersive videos. Simultaneously, data demonstrated that these five types of gratification could influence users’ cognitive engagement with virtual content. In addition, such VR engagement facilitated users’ positive attitudes toward immersive videos and continued usage of them. The findings provided practical implications for COVID-19 global recovery as well.
Industry Applications of Augmented Reality and Virtual Reality

AI, Augmented Reality (AR), Virtual Reality (VIRTUAL REALITY), and also Blockchain is merely a number of the technologies that have affected merely exactly how our provider eat craft, songs, movie, along with exactly how our staff match, shop, in addition to taking a trip. Our pros stay at the center of a total electronic improvement. This paper provides implications and major industry applications of augmented reality and virtual reality.
Automatic detection of head and neck squamous cell carcinoma on histologic slides using hyperspectral microscopic imaging

Automatic, fast, and accurate identification of cancer on histologic slides has many applications in oncologic pathology.The purpose of this study is to investigate hyperspectral imaging (HSI) for automatic detection of head and neck cancer nuclei in histologic slides, as well as cancer region identification based on nuclei detection.A customized hyperspectral microscopic imaging system was developed and used to scan histologic slides from 20 patients with squamous cell carcinoma (SCC). Hyperspectral images and red, green, and blue (RGB) images of the histologic slides with the same field of view were obtained and registered. A principal component analysis-based nuclei segmentation method was developed to extract nuclei patches from the hyperspectral images and the coregistered RGB images. Spectra-based support vector machine and patch-based convolutional neural networks (CNNs) were implemented for nuclei classification. The CNNs were trained with RGB patches (RGB-CNN) and hyperspectral patches (HSI-CNN) of the segmented nuclei and the utility of the extra spectral information provided by HSI was evaluated. Furthermore, cancer region identification was implemented by image-wise classification based on the percentage of cancerous nuclei detected in each image.RGB-CNN, which mainly used the spatial information of nuclei, resulted in a 0.81 validation accuracy and 0.74 testing accuracy. HSI-CNN, which utilized the spatial and spectral features of the nuclei, showed significant improvement in classification performance and achieved 0.89 validation accuracy as well as 0.82 testing accuracy. Furthermore, the image-wise cancer region identification based on nuclei detection could generally improve the cancer detection rate.We demonstrated that the morphological and spectral information contribute to SCC nuclei differentiation and that the spectral information within hyperspectral images could improve classification performance.
Lifelog Retrieval From Daily Digital Data: Narrative Review

Background Over the past decade, the wide availability and small size of different types of sensors, together with the decrease in pricing, have allowed the acquisition of a substantial amount of data about a person’s life in real time. These sensors can be incorporated into personal electronic devices available at a reasonable cost, such as smartphones and small wearable devices. They allow the acquisition of images, audio, location, physical activity, and physiological signals among other data. With these data, usually denoted as lifelog data, we can then analyze and understand personal experiences and behaviors. This process is called lifelogging. Objective The objective of this paper was to present a narrative review of the existing literature about lifelogging over the past decade. To achieve this goal, we analyzed lifelogging applications used to retrieve relevant information from daily digital data, some of them with the purpose of monitoring and assisting people with memory issues and others designed for memory augmentation. We aimed for this review to be used by researchers to obtain a broad idea of the type of data used, methodologies, and applications available in this research field. Methods We followed a narrative review methodology to conduct a comprehensive search for relevant publications in Google Scholar and Scopus databases using lifelog topic–related keywords. A total of 411 publications were retrieved and screened. Of these 411 publications, 114 (27.7%) publications were fully reviewed. In addition, 30 publications were manually included based on our bibliographical knowledge of this research field. Results From the 144 reviewed publications, a total of 113 (78.5%) were selected and included in this narrative review based on content analysis. The findings of this narrative review suggest that lifelogs are prone to become powerful tools to retrieve memories or increase knowledge about an individual’s experiences or behaviors. Several computational tools are already available for a considerable range of applications. These tools use multimodal data of different natures, with visual lifelogs being one of the most used and rich sources of information. Different approaches and algorithms to process these data are currently in use, as this review will unravel. Moreover, we identified several open questions and possible lines of investigation in lifelogging. Conclusions The use of personal lifelogs can be beneficial to improve the quality of our life, as they can serve as tools for memory augmentation or for providing support to people with memory issues. Through the acquisition and analysis of lifelog data, lifelogging systems can create digital memories that can be potentially used as surrogate memory. Through this narrative review, we understand that contextual information can be extracted from lifelogs, which provides an understanding of the daily life of a person based on events, experiences, and behaviors.
AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks

Transformer-based pre-trained models with millions of parameters require large storage.Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters.In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed.AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer.Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias.The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with fine-tuned pretrained models.We further find that Adapter-Bias automatically learns to assign more significant representation shifts to the tokens related to the task in consideration. 1