Reviewerly/exHarmony huggingface dataset:
Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection

Neural Architecture Search (NAS) has achieved great success in image classification task. Some recent works have managed to explore the automatic design of efficient backbone or feature fusion layer for object detection. However, these methods focus on searching only one certain component of object detector while leaving others manually designed. We identify the inconsistency between searched component and manually designed ones would withhold the detector of stronger performance. To this end, we propose a hierarchical trinity search framework to simultaneously discover efficient architectures for all components (i.e. backbone, neck, and head) of object detector in an end-to-end manner. In addition, we empirically reveal that different parts of the detector prefer different operators. Motivated by this, we employ a novel scheme to automatically screen different sub search spaces for different components so as to perform the end-to-end search for each component on the corresponding sub search space efficiently. Without bells and whistles, our searched architecture, namely Hit-Detector, achieves 41.4% mAP on COCO minival set with 27M parameters. Our implementation is available at https://github.com/ggjy/HitDet.pytorch.
Blockchain-based secured event-information sharing protocol in internet of vehicles for smart cities

An intelligent transportation system is an advanced application that aims to provide traffic congestion information, road accidents, emergency information to other vehicles, etc. In the traditional transportation systems, the road-side unit (RSU) acts as a central authority and plays an important role to maintain everything in the networks. To solve single point of failure, as well as to achieve data immutability, blockchain-based decentralized vehicular ad-hoc networks is very essential and important. In order to share critical information among all the road-side units, we design a blockchain-based decentralized vehicular ad-hoc network that supports data immutability property. We have also designed an authentication protocol for the vehicle’s user and a consensus mechanism to validate transactions. Besides, a smart contract mechanism has also been proposed. Informal security analysis confirms anonymity property with other security requirements. We have also calculated and discussed communication, computation, and storage overhead of the proposed scheme.
Detecting Adversarial Samples Using Influence Functions and Nearest Neighbors

Deep neural networks (DNNs) are notorious for their vulnerability to adversarial attacks, which are small perturbations added to their input images to mislead their prediction. Detection of adversarial examples is, therefore, a fundamental requirement for robust classification frameworks. In this work, we present a method for detecting such adversarial attacks, which is suitable for any pre-trained neural network classifier. We use influence functions to measure the impact of every training sample on the validation set data. From the influence scores, we find the most supportive training samples for any given validation example. A k-nearest neighbor (k-NN) model fitted on the DNN's activation layers is employed to search for the ranking of these supporting training samples. We observe that these samples are highly correlated with the nearest neighbors of the normal inputs, while this correlation is much weaker for adversarial inputs. We train an adversarial detector using the k-NN ranks and distances and show that it successfully distinguishes adversarial examples, getting state-of-the-art results on six attack methods with three datasets. Code is available at https://github.com/giladcohen/NNIF_adv_defense.
Semantic Correspondence as an Optimal Transport Problem

Establishing dense correspondences across semantically similar images is a challenging task. Due to the large intra-class variation and background clutter, two common issues occur in current approaches. First, many pixels in a source image are assigned to one target pixel, i.e., many to one matching. Second, some object pixels are assigned to the background pixels, i.e., background matching. We solve the first issue by global feature matching, which maximizes the total matching correlations between images to obtain a global optimal matching matrix. The row sum and column sum constraints are enforced on the matching matrix to induce a balanced solution, thus suppressing the many to one matching. We solve the second issue by applying a staircase function on the class activation maps to re-weight the importance of pixels into four levels from foreground to background. The whole procedure is combined into a unified optimal transport algorithm by converting the maximization problem to the optimal transport formulation and incorporating the staircase weights into optimal transport algorithm to act as empirical distributions. The proposed algorithm achieves state-of-the-art performance on four benchmark datasets. Notably, a 26\% relative improvement is achieved on the large-scale SPair-71k dataset.
Rethinking Zero-Shot Video Classification: End-to-End Training for Realistic Applications

Trained on large datasets, deep learning (DL) can accurately classify videos into hundreds of diverse classes. However, video data is expensive to annotate. Zero-shot learning (ZSL) proposes one solution to this problem. ZSL trains a model once, and generalizes to new tasks whose classes are not present in the training dataset. We propose the first end-to-end algorithm for ZSL in video classification. Our training procedure builds on insights from recent video classification literature and uses a trainable 3D CNN to learn the visual features. This is in contrast to previous video ZSL methods, which use pretrained feature extractors. We also extend the current benchmarking paradigm: Previous techniques aim to make the test task unknown at training time but fall short of this goal. We encourage domain shift across training and test data and disallow tailoring a ZSL model to a specific test dataset. We outperform the state-of-the-art by a wide margin. Our code, evaluation procedure and model weights are available online github.com/bbrattoli/ZeroShotVideoClassification.
Rethinking Depthwise Separable Convolutions: How Intra-Kernel Correlations Lead to Improved MobileNets

We introduce blueprint separable convolutions (BSConv) as highly efficient building blocks for CNNs. They are motivated by quantitative analyses of kernel properties from trained models, which show the dominance of correlations along the depth axis. Based on our findings, we formulate a theoretical foundation from which we derive efficient implementations using only standard layers. Moreover, our approach provides a thorough theoretical derivation, interpretation, and justification for the application of depthwise separable convolutions (DSCs) in general, which have become the basis of many modern network architectures. Ultimately, we reveal that DSC-based architectures such as MobileNets implicitly rely on cross-kernel correlations, while our BSConv formulation is based on intra-kernel correlations and thus allows for a more efficient separation of regular convolutions. Extensive experiments on large-scale and fine-grained classification datasets show that BSConvs clearly and consistently improve MobileNets and other DSC-based architectures without introducing any further complexity. For fine-grained datasets, we achieve an improvement of up to 13.7 percentage points. In addition, if used as drop-in replacement for standard architectures such as ResNets, BSConv variants also outperform their vanilla counterparts by up to 9.5 percentage points on ImageNet.
A Blockchain-Based Searchable Public-Key Encryption With Forward and Backward Privacy for Cloud-Assisted Vehicular Social Networks

As the integration of the Internet of Vehicles and social networks, vehicular social networks (VSN) not only improves the efficiency and reliability of vehicular communication environment, but also provide more comprehensive social services for users. However, with the emergence of advanced communication and computing technologies, more and more data can be fast and conveniently collected from heterogeneous devices, and VSN has to meet new security challenges such as data security and privacy protection. Searchable encryption (SE) as a promising cryptographic primitive is devoted to data confidentiality without sacrificing data searchability. However, most existing schemes are vulnerable to the adaptive leakage-exploiting attacks or can not meet the efficiency requirements of practical applications, especially the searchable public-key encryption schemes (SPE). To achieve secure and efficient keyword search in VSN, we design a new blockchain-based searchable public-key encryption scheme with forward and backward privacy (BSPEFB). BSPEFB is a decentralized searchable public-key encryption scheme since the central search cloud server is replaced by the smart contract. Meanwhile, BSPEFB supports forward and backward privacy to achieve privacy protection. Finally, we implement a prototype of our basic construction and demonstrate the practicability of the proposed scheme in applications.
An Efficient Hardware Accelerator for Structured Sparse Convolutional Neural Networks on FPGAs

Deep convolutional neural networks (CNNs) have achieved state-of-the-art performance in a wide range of applications. However, deeper CNN models, which are usually computation consuming, are widely required for complex artificial intelligence (AI) tasks. Though recent research progress on network compression, such as pruning, has emerged as a promising direction to mitigate computational burden, existing accelerators are still prevented from completely utilizing the benefits of leveraging sparsity due to the irregularity caused by pruning. On the other hand, field-programmable gate arrays (FPGAs) have been regarded as a promising hardware platform for CNN inference acceleration. However, most existing FPGA accelerators focus on dense CNN and cannot address the irregularity problem. In this article, we propose a sparsewise dataflow to skip the cycles of processing multiply-and-accumulates (MACs) with zero weights and exploit data statistics to minimize energy through zeros gating to avoid unnecessary computations. The proposed sparsewise dataflow leads to a low bandwidth requirement and high data sharing. Then, we design an FPGA accelerator containing a vector generator module (VGM) that can match the index between sparse weights and input activations according to the proposed dataflow. Experimental results demonstrate that our implementation can achieve 987-, 46-, and 57-imag/s performance for AlexNet, VGG-16, and ResNet-50 on Xilinx ZCU102, respectively, which provides 1.5×-6.7× speedup and 2.0×-6.0× energy efficiency over previous CNN FPGA accelerators.
A Graph Neural Network Framework for Social Recommendations

Data in many real-world applications such as social networks, users shopping behaviors, and inter-item relationships can be represented as graphs. Graph Neural Networks (GNNs) have shown great success in learning meaningful representations for graphs by inherently integrating node information and topological structure. Data in social recommendations can also be denotes as graph data in the form of user-user social graphs and user-item graphs. In addition, the relationships between items can be denoted as item-item graphs. GNNs provide an unprecedented opportunity to advance social recommendations. However, there are tremendous challenges in building GNNs-based social recommendations where (1) users (items) are simultaneously involved in the user-item graph and user-user social graph (item-item graph); (2) user-item graphs not only contain user-item interactions but also include users’ opinions on items; and (3) the nature of social relations are heterogeneous among users. In this paper, we propose a novel graph neural network framework ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GraphRec+</b> ) for social recommendations, which is able to coherently model graph data in order to learn better user and item representations. Specifically, we introduce a principled approach for jointly capturing interactions and opinions in the user-item graph and also propose an attention mechanism to differentiate the heterogeneous strengths of social relations. Comprehensive experiments on three real-world datasets show the effectiveness of the proposed framework.
Functional code clone detection with syntax and semantics fusion learning

Clone detection of source code is among the most fundamental software engineering techniques. Despite intensive research in the past decade, existing techniques are still unsatisfactory in detecting "functional" code clones. In particular, existing techniques cannot efficiently extract syntax and semantics information from source code. In this paper, we propose a novel joint code representation that applies fusion embedding techniques to learn hidden syntactic and semantic features of source codes. Besides, we introduce a new granularity for functional code clone detection. Our approach regards the connected methods with caller-callee relationships as a functionality and the method without any caller-callee relationship with other methods represents a single functionality. Then we train a supervised deep learning model to detect functional code clones. We conduct evaluations on a large dataset of C++ programs and the experimental results show that fusion learning can significantly outperform the state-of-the-art techniques in detecting functional code clones.
Heterogeneous teaching evaluation network based offline course recommendation with graph learning and tensor factorization

Course recommendation systems are applied to help students with different needs select courses in a large range of course resources. However, a student’s needs are not always determined by their personal interests, they are also influenced by teachers, peers etc. Unlike online courses, user behavior and user satisfaction of offline courses often have serious sparse and cold start issues, which cause overfitting problems in previous neural network and matrix factorization (MF) models. Additionally, the interpersonal relations, evaluation text and existing “user-item” formatted rating matrix constitute a multi-source and multi-modal data structure, so a systematic data fusion method is needed to establish recommendations based on these heterogeneous characteristics. Therefore, a hybrid recommendation model by fusing network structured feature with graph neural networks and user interactive activities with tensor factorization was proposed in this paper. First, a graph structured teaching evaluation network is proposed to describe students, courses, and other entities by using the students’ rating, commentary text, grading and interpersonal relations. Then, a random walk based neural network is employed to generate the vectorized representation of students by learning their own relational structure. Finally, by recognizing these personalization features as the third dimension of the rating tensor, a Bayesian Probabilistic Tensor Factorization-based tensor factorization is applied to learn and predict students’ ratings for classes they have not taken. Experiments on a real-world evaluation of teaching system including 532 participants with 7,453 rating records show that the proposed method outperforms other existing neural network and matrix factorization models including xSVD++, RTTF and DSE with a smaller predictive error as well as better recommendation accuracy.
Deep Models and Shortwave Infrared Information to Detect Face Presentation Attacks

This paper addresses the problem of face presentation attack detection using different image modalities. In particular, the usage of short wave infrared (SWIR) imaging is considered. Face presentation attack detection is performed using recent models based on Convolutional Neural Networks using only carefully selected SWIR image differences as input. Conducted experiments show superior performance over similar models acting on either color images or on a combination of different modalities (visible, NIR, thermal and depth), as well as on a SVM-based classifier acting on SWIR image differences. Experiments have been carried on a new public and freely available database, containing a wide variety of attacks. Video sequences have been recorded thanks to several sensors resulting in 14 different streams in the visible, NIR, SWIR and thermal spectra, as well as depth data. The best proposed approach is able to almost perfectly detect all impersonation attacks while ensuring low bonafide classification errors. On the other hand, obtained results show that obfuscation attacks are more difficult to detect. We hope that the proposed database will foster research on this challenging problem. Finally, all the code and instructions to reproduce presented experiments is made available to the research community.
IoT based Clinical Sensor Data Management and Transfer using Blockchain Technology

There has been revolutionary developments in the healthcare industry with the advancement of technology over the past years. Internet of Things, Cloud Computing, Blockchain technology, lab-on-chip, non-invasive and minimally invasive surgeries and so on has simplified several dreadful diseases. The research as well as healthcare industry have been greatly impacted by these new technologies. Clinical exams and self-health tracking can be done by means of miniaturized healthcare sensors that are powered by IoT. They help in early diagnosis and treatment guidance by clinicians at remote locations without directly being in contact with the users. The access control structures and inconsistent security policies have been a hinderance in meeting the security requirements of these data. Blockchain based smart contracts and enterprise-distributed ledger framework can be used for monitoring the vital signs of the patient. This enables accessing medical information of patients globally at any time along with immutable and extensive history log. In comparison with the traditional patient monitoring system, the proposed system offers better monitoring, improved connectivity and enhanced data security.
FTRANS

In natural language processing (NLP), the "Transformer" architecture was proposed as the first transduction model replying entirely on self-attention mechanisms without using sequence-aligned recurrent neural networks (RNNs) or convolution, and it achieved significant improvements for sequence to sequence tasks. The introduced intensive computation and storage of these pre-trained language representations has impeded their popularity into computation and memory constrained devices. The field-programmable gate array (FPGA) is widely used to accelerate deep learning algorithms for its high parallelism and low latency. However, the trained models are still too large to accommodate to an FPGA fabric. In this paper, we propose an efficient acceleration framework, Ftrans, for transformer-based large scale language representations. Our framework includes enhanced block-circulant matrix (BCM)-based weight representation to enable model compression on large-scale language representations at the algorithm level with few accuracy degradation, and an acceleration design at the architecture level. Experimental results show that our proposed framework significantly reduce the model size of NLP models by up to 16 times. Our FPGA design achieves 27.07× and 81 × improvement in performance and energy efficiency compared to CPU, and up to 8.80× improvement in energy efficiency compared to GPU.
Depth-Quality-Aware Salient Object Detection

The existing fusion based RGB-D salient object detection methods usually adopt the bi-stream structure to strike the fusion trade-off between RGB and depth (D). The D quality usually varies from scene to scene, while the SOTA bi-stream approaches are depth quality unaware, which easily result in substantial difficulties in achieving complementary fusion status between RGB and D, leading to poor fusion results in facing of low-quality D. Thus, this paper attempts to integrate a novel depth quality aware subnet into the classic bi-stream structure, aiming to assess the depth quality before conducting the selective RGB-D fusion. Compared with the SOTA bi-stream methods, the major highlight of our method is its ability to lessen the importance of those low-quality, no-contribution, or even negative-contribution D regions during the RGB-D fusion, achieving a much improved complementary status between RGB and D.
A framework for anomaly detection and classification in Multiple IoT scenarios

The investigation of anomalies is an important element in many scientific research fields. In recent years, this activity has been also extended to social networking and social internetworking, where different networks interact with each other. In these research fields, we have recently witnessed an important evolution because, beside networks of people, networks of things are becoming increasingly common. IoT and Multiple IoT scenarios are thus more and more studied. This paper represents a first attempt to investigate anomalies in a Multiple IoT scenario (MIoT). First, we propose a new methodological framework that can make future investigations in this research field easier, coherent, and uniform. Then, in the context of anomaly detection in an MIoT, we define the so-called “forward problem” and “inverse problem”. The definition of these problems allows the investigation of how anomalies depend on inter-node distances, the size of IoT networks, and the degree centrality and closeness centrality of anomalous nodes. The approach proposed herein is applied to a smart city scenario, which is a typical MIoT. Here, data coming from sensors and social networks can boost smart lighting in order to provide citizens with a smart and safe environment.
FedRec: Federated Recommendation With Explicit Feedback

Recommendation models have been widely embedded in various online services, while most of which are designed with the assumption that users’ original behaviors are available in a central server. This may cause the privacy issue. As a response, we follow a recent work called federated collaborative filtering (FCF) for item recommendation with implicit feedback and propose a novel and generic federated recommendation (FedRec) framework for rating prediction with explicit feedback. Specifically, we federate some basic and advanced factorization-based recommendation models both in batch style and in stochastic style. More importantly, in order to protect the private information of which items each user has rated, as well as not to significantly increase the computational and communication cost, we design two simple but effective strategies, i.e., user averaging and hybrid filling, in which some (instead of all) unrated items are randomly sampled and assigned with some virtual ratings accordingly. Empirical studies on two public datasets show the effectiveness of our FedRec in terms of the closeness of a federated model and an unfederated one, and the usefulness of the two filling strategies.