Reviewerly/exHarmony huggingface dataset:
IoT based Assistive Device for Deaf, Dumb and Blind People

Focusing and addressing the problems faced by the differently abled people such as visually, audibly and vocally challenged, through a single device is a tough job. A lot of research has been done on each problem and solutions have been proposed separately. But not all of them are addressed together. The aim of the project is to create a single device solution in such a way that is simple, fast, accurate and cost-effective. The main purpose of the device is to make the differently abled people, feel independent confident by seeing, hearing and talking for them. The paper provides a Google API and Raspberry Pi based aid for the blind deaf and dumb people. The proposed device enables visually challenged people to read by taking an image. Further, Image to text conversion and speech synthesis is done, converting it into an audio format that reads out the extracted text translating documents, books and other available materials in daily life. For the audibly challenged, the input is in form of speech taken in by the microphone and recorded audio is then converted into text which is displayed in the form of a pop-up window for the user in the screen of the device. The vocally impaired are aided by taking the input by the user as text through the built-in customized on-screen keyboard where the text is identified, text into speech conversion is done and the speaker gives the speech output. This way the device speaks for the user.
MLSL: Multi-Level Self-Supervised Learning for Domain Adaptation with Spatially Independent and Semantically Consistent Labeling

Most of the recent Deep Semantic Segmentation algorithms suffer from large generalization errors, even when powerful hierarchical representation models, based on convolutional neural networks, have been employed. This could be attributed to limited training data and large distribution gap in train and test domain datasets. In this paper, we propose a multi-level self-supervised learning model for domain adaptation of semantic segmentation. Exploiting the idea that an object (and most of the stuff given context) should be labeled consistently regardless of its location, we generate spatially independent and semantically consistent (SISC) pseudo-labels by segmenting multiple sub-images using base model and designing an aggregation strategy. Image level pseudo weak-labels, PWL, are computed to guide domain adaptation by capturing global context similarity in source and target domain at latent space level. Thus helping latent space learn the representation even when there are very few pixels belonging to the domain category (small object for example) compared to rest of the image. Our multi-level Self-supervised learning (MLSL) outperforms existing state-of-art (self or adversarial learning) algorithms. Specifically, keeping all setting similar and employing MLSL we obtain an mIoUgain of 5.1% on GTA-V to Cityscapes adaptation and 4.3% on SYNTHIA to Cityscapes adaptation compared to existing state-of-art method.
Talking With Hands 16.2M: A Large-Scale Dataset of Synchronized Body-Finger Motion and Audio for Conversational Motion Analysis and Synthesis

We present a 16.2-million frame (50-hour) multimodal dataset of two-person face-to-face spontaneous conversations. Our dataset features synchronized body and finger motion as well as audio data. To the best of our knowledge, it represents the largest motion capture and audio dataset of natural conversations to date. The statistical analysis verifies strong intraperson and interperson covariance of arm, hand, and speech features, potentially enabling new directions on data-driven social behavior analysis, prediction, and synthesis. As an illustration, we propose a novel real-time finger motion synthesis method: a temporal neural network innovatively trained with an inverse kinematics (IK) loss, which adds skeletal structural information to the generative model. Our qualitative user study shows that the finger motion generated by our method is perceived as natural and conversation enhancing, while the quantitative ablation study demonstrates the effectiveness of IK loss.
Highly dispersive substitution box (S‐box) design using chaos

Highly dispersive S-boxes are desirable in cryptosystems as nonlinear confusion sub-layers for resisting modern attacks. For a near optimal cryptosystem resistant to modern cryptanalysis, a highly nonlinear and low differential probability (DP) value is required. We propose a method based on a piecewise linear chaotic map (PWLCM) with optimization conditions. Thus, the linear propagation of information in a cryptosystem appearing as a high DP during differential cryptanalysis of an S-box is minimized. While mapping from the chaotic trajectory to integer domain, a randomness test is performed that justifies the nonlinear behavior of the highly dispersive and nonlinear chaotic S-box. The proposed scheme is vetted using well-established cryptographic performance criteria. The proposed S-box meets the cryptographic performance criteria and further minimizes the differential propagation justified by the low DP value. The suitability of the proposed S-box is also tested using an image encryption algorithm. Results show that the proposed S-box as a confusion component entails a high level of security and improves resistance against all known attacks.
Variational Image Deraining

Images captured in severe weather such as rain and snow significantly degrade the accuracy of vision systems, e.g., for outdoor video surveillance or autonomous driving. Image deraining is a critical yet highly challenging task, due to the fact that rain density varies across spatial locations, while the distribution patterns simultaneously vary across color channels. In this paper, we propose a variational image deraining (VID) method by formulating image deraining in a conditional variational auto-encoder framework. To achieve adaptive deraining to spatial rain density, we generate a density estimation map for each color channel, which can largely avoid over and under deraining. In addition, to address cross-channel variations, we conduct channel-wise deraining, motivated by our observation that bright pixels do not tend to remain bright after deraining unless their color channels are handled separately. Experimental results show that the proposed deraining method achieves superior performance on both synthesized and real rainy images, surpassing previous state-of-the-art methods by large margins.
LUTNet: Learning FPGA Configurations for Highly Efficient Neural Network Inference

Research has shown that deep neural networks contain significant redundancy, and thus that high classification accuracy can be achieved even when weights and activations are quantized down to binary values. Network binarization on FPGAs greatly increases area efficiency by replacing resource-hungry multipliers with lightweight XNOR gates. However, an FPGA's fundamental building block, the K-LUT, is capable of implementing far more than an XNOR: it can perform any K-input Boolean operation. Inspired by this observation, we propose LUTNet, an end-to-end hardware-software framework for the construction of area-efficient FPGA-based neural network accelerators using the native LUTs as inference operators. We describe the realization of both unrolled and tiled LUTNet architectures, with the latter facilitating smaller, less power-hungry deployment over the former while sacrificing area and energy efficiency along with throughput. For both varieties, we demonstrate that the exploitation of LUT flexibility allows for far heavier pruning than possible in prior works, resulting in significant area savings while achieving comparable accuracy. Against the state-of-the-art binarized neural network implementation, we achieve up to twice the area efficiency for several standard network models when inferencing popular datasets. We also demonstrate that even greater energy efficiency improvements are obtainable.
Real-Time Object Detection Using Pre-Trained Deep Learning Models MobileNet-SSD

Mobile networks and binary neural networks are the most commonly used techniques for modern deep learning models to perform a variety of tasks on embedded systems. In this paper, we develop a technique to identify an object considering the deep learning pre-trained model MobileNet for Single Shot Multi-Box Detector (SSD). This algorithm is used for real-time detection, and for webcam feed to detect the purpose webcam which detects the object in a video stream. Therefore, we use an object detection module that can detect what is in the video stream. In order to implement the module, we combine the MobileNet and the SSD framework for a fast and efficient deep learning-based method of object detection. The main purpose of our research is to elaborate the accuracy of an object detection method SSD and the importance of pre-trained deep learning model MobileNet. The experimental results show that the Average Precision (AP) of the algorithm to detect different classes as car, person and chair is 99.76%, 97.76% and 71.07%, respectively. This improves the accuracy of behavior detection at a processing speed which is required for the real-time detection and the requirements of daily monitoring indoor and outdoor.
DeepSurvNet: deep survival convolutional network for brain cancer survival rate classification based on histopathological images

Histopathological whole slide images of haematoxylin and eosin (H&E)-stained biopsies contain valuable information with relation to cancer disease and its clinical outcomes. Still, there are no highly accurate automated methods to correlate histolopathological images with brain cancer patients' survival, which can help in scheduling patients therapeutic treatment and allocate time for preclinical studies to guide personalized treatments. We now propose a new classifier, namely, DeepSurvNet powered by deep convolutional neural networks, to accurately classify in 4 classes brain cancer patients' survival rate based on histopathological images (class I, 0-6 months; class II, 6-12 months; class III, 12-24 months; and class IV, >24 months survival after diagnosis). After training and testing of DeepSurvNet model on a public brain cancer dataset, The Cancer Genome Atlas, we have generalized it using independent testing on unseen samples. Using DeepSurvNet, we obtained precisions of 0.99 and 0.8 in the testing phases on the mentioned datasets, respectively, which shows DeepSurvNet is a reliable classifier for brain cancer patients' survival rate classification based on histopathological images. Finally, analysis of the frequency of mutations revealed differences in terms of frequency and type of genes associated to each class, supporting the idea of a different genetic fingerprint associated to patient survival. We conclude that DeepSurvNet constitutes a new artificial intelligence tool to assess the survival rate in brain cancer. Graphical abstract A DCNN model was generated to accurately predict survival rates of brain cancer patients (classified in 4 different classes) accurately. After training the model using images from H&E stained tissue biopsies from The Cancer Genome Atlas database (TCGA, left), the model can predict for each patient, based on a histological image (top right), its survival class accurately (bottom right).
Multiple Object Forecasting: Predicting Future Object Locations in Diverse Environments

This paper introduces the problem of multiple object forecasting (MOF), in which the goal is to predict future bounding boxes of tracked objects. In contrast to existing works on object trajectory forecasting which primarily consider the problem from a birds-eye perspective, we formulate the problem from an object-level perspective and call for the prediction of full object bounding boxes, rather than trajectories alone. Towards solving this task, we introduce the Citywalks dataset, which consists of over 200k high-resolution video frames. Citywalks comprises of footage recorded in 21 cities from 10 European countries in a variety of weather conditions and over 3.5k unique pedestrian trajectories. For evaluation, we adapt existing trajectory forecasting methods for MOF and confirm cross-dataset generalizability on the MOT-17 dataset without fine-tuning. Finally, we present STED, a novel encoder-decoder architecture for MOF. STED combines visual and temporal features to model both object-motion and ego-motion, and outperforms existing approaches for MOF. Code & dataset link: https://github.com/olly-styles/Multiple-Object-Forecasting
View Independent Generative Adversarial Network for Novel View Synthesis

Synthesizing novel views from a 2D image requires to infer 3D structure and project it back to 2D from a new viewpoint. In this paper, we propose an encoder-decoder based generative adversarial network VI-GAN to tackle this problem. Our method is to let the network, after seeing many images of objects belonging to the same category in different views, obtain essential knowledge of intrinsic properties of the objects. To this end, an encoder is designed to extract view-independent feature that characterizes intrinsic properties of the input image, which includes 3D structure, color, texture etc. We also make the decoder hallucinate the image of a novel view based on the extracted feature and an arbitrary user-specific camera pose. Extensive experiments demonstrate that our model can synthesize high-quality images in different views with continuous camera poses, and is general for various applications.
DeepFuse: An IMU-Aware Network for Real-Time 3D Human Pose Estimation from Multi-View Image

In this paper, we propose a two-stage fully 3D network, namely DeepFuse, to estimate human pose in 3D space by fusing body-worn Inertial Measurement Unit (IMU) data and multi-view images deeply. The first stage is designed for pure vision estimation. To preserve data primitiveness of multi-view inputs, the vision stage uses multi-channel volume as data representation and 3D soft-argmax as activation layer. The second one is the IMU refinement stage which introduces an IMU-bone layer to fuse the IMU and vision data earlier at data level. without requiring a given skeleton model a priori, we can achieve a mean joint error of 28.9mm on TotalCapture dataset and 13.4mm on Human3.6M dataset under protocol 1, improving the SOTA result by a large margin. Finally, we discuss the effectiveness of a fully 3D network for 3D pose estimation experimentally which may benefit future research.
Dynamic Motion Representation for Human Action Recognition

Despite the advances in Human Activity Recognition, the ability to exploit the dynamics of human body motion in videos has yet to be achieved. In numerous recent works, researchers have used appearance and motion as independent inputs to infer the action that is taking place in a specific video. In this paper, we highlight that while using a novel representation of human body motion, we can benefit from appearance and motion simultaneously. As a result, better performance of action recognition can be achieved. We start with a pose estimator to extract the location and heat-map of body joints in each frame. We use a dynamic encoder to generate a fixed size representation from these body joint heat-maps. Our experimental results show that training a convolutional neural network with the dynamic motion representation outperforms state-of-the-art action recognition models. By modeling distinguishable activities as distinct dynamical systems and with the help of two stream networks, we obtain the best performance on HMDB, JHMDB, UCF-101, and AVA datasets.
VCNet: A Robust Approach to Blind Image Inpainting

Blind inpainting is a task to automatically complete visual contents without specifying masks for missing areas in an image. Previous work assumes known missing-region-pattern, limiting the application scope. We instead relax the assumption by defining a new blind inpainting setting, making training a neural system robust against various unknown missing region patterns. Specifically, we propose a two-stage visual consistency network (VCN) to estimate where to fill (via masks) and generate what to fill. In this procedure, the unavoidable potential mask prediction errors lead to severe artifacts in the subsequent repairing. To address it, our VCN predicts semantically inconsistent regions first, making mask prediction more tractable. Then it repairs these estimated missing regions using a new spatial normalization, making VCN robust to mask prediction errors. Semantically convincing and visually compelling content can be generated. Extensive experiments show that our method is effective and robust in blind image inpainting. And our VCN allows for a wide spectrum of applications.
Distribution of trace elements between carbonaceous matter and sulfides in a sediment-hosted orogenic gold system

Carbonaceous matter (CM) plays a critical role in the formation of sediment-hosted ore deposits, but metal partitioning between CM and associated sulfides remains unclear. Here we use synchrotron X-ray fluorescence microscopy (SXRF), laser ablation–inductively coupled plasma–mass spectrometry (LA–ICP–MS), and nanoscale secondary ion mass spectrometry (NanoSIMS) to characterize the distribution and deportment of trace elements between CM and sulfides from the Daqiao orogenic gold deposit, China. Four types of CM co-existing with different generations of sulfides are recognized: sedimentary CM1 with pyrite polyframboids in black shale host; indigenous CM2 with early- to main-ore pyrite and marcasite; CM3 with main-ore pyrite; CM4 with pyrite porphyroblasts in underlying graphitic schists. The SXRF results reveal that As, Se, Ni, Pb, and Cu are enriched in pyrite polyframboids relative to CM1 in black shales, although a small proportion of As, Pb, Bi, and U is accommodated in the latter. NanoSIMS images show that Au and As are associated with the CM1 matrix within pyrite framboids that are overgrown by recrystallized pyrite rims. Most trace elements are enriched in the newly-formed aggregates of pyrite and marcasite relative to the CM2 rims that formed via the interactions between ore fluids and pre-existing CM2 cores. Gold, As, Tl, Se, and Hg are highly enriched in the main-ore pyrite relative to co-precipitated CM3 veinlets, and this deportment might reflect changes in fluid redox conditions in response to hydraulic fracturing that is the main mechanism of gold deposition. The pyrite porphyroblasts in the schists are enriched in As, Se, Ni, and Cu relative to CM4, but have lower Au, As, Co, Ni, and Pb than sedimentary pyrite framboids. Taken together, we suggest that biological activity causes accumulation of As and Au within CM interstitial to the framboidal pyrite, and that these elements are released during metamorphic recrystallization and hydrothermal replacement. Trace element deportment in hydrothermal CM and sulfides is controlled by a variety of factors such as the fluid chemistry and interaction between indigenous CM and fluids. Our quantitative results provide significant new insights into the complex processes that formed carbonaceous sediment-hosted ore systems worldwide.
Attacks on the Industrial Internet of Things – Development of a multi-layer Taxonomy

The Industrial Internet of Things (IIoT) provides new opportunities to improve process and production efficiency, which enable new business models. At the same time, the high degree of cross-linking and decentralization increases the complexity of IIoT systems and creates new vulnerabilities. Hence, organizations are not only vulnerable to conventional IT threats, but also to a multitude of new, IIoT-specific attacks. Yet, a literature-based and empirically evaluated understanding of attacks on the IIoT is still lacking. Against this backdrop, we develop a multi-layer taxonomy that helps researchers and practitioners to identify similarities and differences between attacks on the IIoT. Based on the latest literature and a sample of about 50 attacks, we deductively and inductively determine attack characteristics and dimensions. We demonstrate the usefulness and practical relevance of our taxonomy by applying it to a real-world incident affecting a German steel facility. By combining IT security, IIoT, and risk management to form an interdisciplinary approach, we contribute to the descriptive knowledge in these fields. Industry experts confirm that our taxonomy enables a detailed classification of attacks, which supports the identification, documentation, and communication of incidents within organizations and their value-creation networks. With this, our taxonomy provides a profound basis for the further development of IT security management and the derivation of mitigation measures.
A multi-level fusion based decision support system for academic collaborator recommendation

In academia, researchers collaborate with their peers to improve the quality of research and thereby enhance academic profiles. However, information overload in big scholarly data poses a challenge in identifying potential researchers for fruitful collaboration. In this article, we introduce a multi-level fusion-based model for collaborator recommendation, DRACoR (Deep learning and Random walk based Academic Collaborator Recommender). DRACoR fuses deep learning and biased random walk model to provide the recommendation for potential collaborators that share similar research interests at the peer level. We run a topic model on abstracts and Doc2Vec on titles on year-wise publications to capture the dynamic research interests of researchers. Author–author cosine similarity is computed from the feature vectors extracted from abstracts and titles and is then used to weigh edges in the author–author graph (AAG). We also aggregate various meta-path features with profile-aware features to bias the random walk behavior. Finally, we employ a random walk with restart(RWR) to recommend top N collaborators where the edge weights are used to bias the random walker's behavior. Extensive experiments on DBLP and hep-th datasets demonstrate the effectiveness of our proposed DRACoR model against various state-of-the-art methods in terms of precision, recall, F1-score, MRR, and nDCG.
Probabilistic color visual cryptography schemes for black and white secret images

Color-black-and-white visual cryptography scheme (CBW-VCS) is a methodology that utilizes colors to alleviate the pixel expansion problem. In a general (k,n) CBW-VCS, when k and n become larger, the pixel expansion increases dramatically. In this paper, two constructions for constituting a (k,n) threshold probabilistic CBW-VCS (PCBW-VCS) are introduced, where the generated color shares are non-expansible. The two proposed constructions are proven to be valid constructions which satisfy the security and contrast conditions. Theoretical analysis and sufficient experiments are demonstrated to shown the effectiveness and advantages of the proposed PCBW-VCSs.