Reviewerly/exHarmony huggingface dataset:
DeepShift: Towards Multiplication-Less Neural Networks

The high computation, memory, and power budgets of inferring convolutional neural networks (CNNs) are major bottlenecks of model deployment to edge computing platforms, e.g., mobile devices and IoT. Moreover, training CNNs is time and energy-intensive even on high-grade servers. Convolution layers and fully connected layers, because of their intense use of multiplications, are the dominant contributor to this computation budget.We propose to alleviate this problem by introducing two new operations: convolutional shifts and fully-connected shifts which replace multiplications with bitwise shift and sign flipping during both training and inference. During inference, both approaches require only 5 bits (or less) to represent the weights. This family of neural network architectures (that use convolutional shifts and fully connected shifts) is referred to as DeepShift models. We propose two methods to train DeepShift models: DeepShift-Q which trains regular weights constrained to powers of 2, and DeepShift-PS that trains the values of the shifts and sign flips directly.Very close accuracy, and in some cases higher accuracy, to baselines are achieved. Converting pre-trained 32-bit floating-point baseline models of ResNet18, ResNet50, VGG16, and GoogleNet to DeepShift and training them for 15 to 30 epochs, resulted in Top-1/Top-5 accuracies higher than that of the original model. Last but not least, we implemented the convolutional shifts and fully connected shift GPU kernels and showed a reduction in latency time of 25% when inferring ResNet18 compared to unoptimized multiplication-based GPU kernels. The code can be found at https://github.com/mostafaelhoushi/DeepShift.
Logish: A new nonlinear nonmonotonic activation function for convolutional neural network

Activation function is an important component of the convolutional neural network. Recently, nonlinear nonmonotonic activation functions such as Swish and Mish have illustrated good performance in deep learning structures. In this paper, we propose a new nonlinear nonmonotonic activation function called Logish, which can be represented by f(x)=x·ln[1+sigmoid(x)]. Firstly, we take the logarithmic operation to reduce the numerical range of sigmoid(x)+1, then we employ variable x to make the negative output have a strong regularization effect. Furthermore, we evaluate the image classification performance of Logish and its variant f(x)=αx·ln[1+sigmoid(βx)] in simple and complex networks with top–1 accuracy. Experimental results demonstrate that Logish’s variant (α=1,β=10) can achieve 94.8% top-1 accuracy with ResNet–50 network on CIFAR 10 dataset, and can reach 99.24% top-1 accuracy with DenseNet on MNIST dataset and 88.52% top-1 accuracy with SE-Inception-v4 network on SVHN dataset respectively. It is higher than the Sigmoid, Tanh, ReLU, Swish and Mish activation functions in the corresponding dataset. It also verifies the performance and effectiveness of Logish.
Dogfight: Detecting Drones from Drones Videos

As airborne vehicles are becoming more autonomous and ubiquitous, it has become vital to develop the capability to detect the objects in their surroundings. This paper attempts to address the problem of drones detection from other flying drones. The erratic movement of the source and target drones, small size, arbitrary shape, large intensity variations, and occlusion make this problem quite challenging. In this scenario, region-proposal based methods are not able to capture sufficient discriminative foreground-background information. Also, due to the extremely small size and complex motion of the source and target drones, feature aggregation based methods are unable to perform well. To handle this, instead of using region-proposal based methods, we propose to use a two-stage segmentation-based approach employing spatio-temporal attention cues. During the first stage, given the overlapping frame regions, detailed contextual information is captured over convolution feature maps using pyramid pooling. After that pixel and channel-wise attention is enforced on the feature maps to ensure accurate drone localization. In the second stage, first stage detections are verified and new probable drone locations are explored. To discover new drone locations, motion boundaries are used. This is followed by tracking candidate drone detections for a few frames, cuboid formation, extraction of the 3D convolution feature map, and drones detection within each cuboid. The proposed approach is evaluated on two publicly available drone detection datasets and outperforms several competitive baselines.
DVFENet: Dual-branch voxel feature extraction network for 3D object detection

3D object detection based on LiDAR point cloud has wide applications in autonomous driving and robotics. Recently, many approaches use voxelization representation in feature extraction and apply 3D convolution neural networks for 3D object detection. How to get expressive 3D voxelization representation is important for the detection performance. Therefore, we propose a new 3D object detection framework (DVFENet) based on dual-branch voxel feature extraction, which can provide rich and complete 3D information. The first branch is a graph-attention-network-based voxel feature extraction, which applies an improved voxel graph attention feature extractor (VGAFE) on large-scale voxelization. This branch uses graph convolution networks with an attention mechanism to extract more local neighborhood and context information. The second branch is a 3D-sparse-convolution-based voxel feature extraction that captures finer geometric features based on small-scale voxelization. We also design a decoupled RPN module that can obtain task-specific features to reduce the task conflict. Experiments on the challenging KITTI 3D object detection benchmark and nuScenes detection task show that our method achieve good performance. At the same time, we conduct extensive experiments to verify the effectiveness of each component.
‘Welcome to #GabFam’: Far-right virtual community on Gab

With large social media platforms coming under increasing pressure to deplatform far-right users, the Alternative Technology movement (Alt-Tech) emerged as a new digital support infrastructure for the far right. We conduct a qualitative analysis of the prominent Alt-Tech platform Gab, a social networking service primarily modelled on Twitter, to assess the far-right virtual community on the platform. We find Gab’s technological affordances – including its lack of content moderation, culture of anonymity, microblogging architecture and funding model – have fostered an ideologically eclectic far-right community united by fears of persecution at the hands of ‘Big Tech’. We argue that this points to the emergence of a novel techno-social victimology as an axis of far-right virtual community, wherein shared experiences or fears of being deplatformed facilitate a coalescing of assorted far-right tendencies online.
Invisible Perturbations: Physical Adversarial Examples Exploiting the Rolling Shutter Effect

Physical adversarial examples for camera-based computer vision have so far been achieved through visible artifacts — a sticker on a Stop sign, colorful borders around eyeglasses or a 3D printed object with a colorful texture. An implicit assumption here is that the perturbations must be visible so that a camera can sense them. By contrast, we contribute a procedure to generate, for the first time, physical adversarial examples that are invisible to human eyes. Rather than modifying the victim object with visible artifacts, we modify light that illuminates the object. We demonstrate how an attacker can craft a modulated light signal that adversarially illuminates a scene and causes targeted misclassifications on a state-of-the-art ImageNet deep learning model. Concretely, we exploit the radiometric rolling shutter effect in commodity cameras to create precise striping patterns that appear on images. To human eyes, it appears like the object is illuminated, but the camera creates an image with stripes that will cause ML models to output the attacker-desired classification. We conduct a range of simulation and physical experiments with LEDs, demonstrating targeted attack rates up to 84%.
Weakly Supervised Action Selection Learning in Video

Localizing actions in video is a core task in computer vision. The weakly supervised temporal localization problem investigates whether this task can be adequately solved with only video-level labels, significantly reducing the amount of expensive and error-prone annotation that is required. A common approach is to train a frame-level classifier where frames with the highest class probability are selected to make a video-level prediction. Frame-level activations are then used for localization. However, the absence of frame-level annotations cause the classifier to impart class bias on every frame. To address this, we propose the Action Selection Learning (ASL) approach to capture the general concept of action, a property we refer to as "actionness". Under ASL, the model is trained with a novel class-agnostic task to predict which frames will be selected by the classifier. Empirically, we show that ASL outperforms leading baselines on two popular bench-marks THUMOS-14 and ActivityNet-1.2, with 10.3% and 5.7% relative improvement respectively. We further analyze the properties of ASL and demonstrate the importance of actionness. Full code for this work is available here: https://github.com/layer6ai-labs/ASL.
Intelligent Carpet: Inferring 3D Human Pose from Tactile Signals

Daily human activities, e.g., locomotion, exercises, and resting, are heavily guided by the tactile interactions between the human and the ground. In this work, leveraging such tactile interactions, we propose a 3D human pose estimation approach using the pressure maps recorded by a tactile carpet as input. We build a low-cost, high-density, large-scale intelligent carpet, which enables the real-time recordings of human-floor tactile interactions in a seamless manner. We collect a synchronized tactile and visual dataset on various human activities. Employing a state-of-the-art camera-based pose estimation model as supervision, we design and implement a deep neural network model to infer 3D human poses using only the tactile information. Our pipeline can be further scaled up to multi-person pose estimation. We evaluate our system and demonstrate its potential applications in diverse fields.
Strengthen Learning Tolerance for Weakly Supervised Object Localization

Weakly supervised object localization (WSOL) aims at learning to localize objects of interest by only using the image-level labels as the supervision. While numerous efforts have been made in this field, recent approaches still suffer from two challenges: one is the part domination issue while the other is the learning robustness issue. Specifically, the former makes the localizer prone to the local discriminative object regions rather than the desired whole object, and the latter makes the localizer over-sensitive to the variations of the input images so that one can hardly obtain localization results robust to the arbitrary visual stimulus. To solve these issues, we propose a novel framework to strengthen the learning tolerance, referred to as SLT-Net, for WSOL. Specifically, we consider two-fold learning tolerance strengthening mechanisms. One is the semantic tolerance strengthening mechanism, which allows the localizer to make mistakes for classifying similar semantics so that it will not concentrate too much on the discriminative local regions. The other is the visual stimuli tolerance strengthening mechanism, which enforces the localizer to be robust to different image transformations so that the prediction quality will not be sensitive to each specific input image. Finally, we implement comprehensive experimental comparisons on two widely-used datasets CUB and ILSVRC2012, which demonstrate the effectiveness of our proposed approach.
ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search

Human pose estimation has achieved significant progress in recent years. However, most of the recent methods focus on improving accuracy using complicated models and ignoring real-time efficiency. To achieve a better trade-off between accuracy and efficiency, we propose a novel neural architecture search (NAS) method, termed ViP-NAS, to search networks in both spatial and temporal levels for fast online video pose estimation. In the spatial level, we carefully design the search space with five different dimensions including network depth, width, kernel size, group number, and attentions. In the temporal level, we search from a series of temporal feature fusions to optimize the total accuracy and speed across multiple video frames. To the best of our knowledge, we are the first to search for the temporal feature fusion and automatic computation allocation in videos. Extensive experiments demonstrate the effectiveness of our approach on the challenging COCO2017 and PoseTrack2018 datasets. Our discovered model family, S-ViPNAS and T-ViPNAS, achieve significantly higher inference speed (CPU real-time) without sacrificing the accuracy compared to the previous state-of-the-art methods.
End-to-End Learning for Joint Image Demosaicing, Denoising and Super-Resolution

Image denoising, demosaicing and super-resolution are key problems of image restoration well studied in the recent decades. Often, in practice, one has to solve these problems simultaneously. A problem of finding a joint solution of the multiple image restoration tasks just begun to attract an increased attention of researchers. In this paper, we propose an end-to-end solution for the joint demosaicing, denoising and super-resolution based on a specially designed deep convolutional neural network (CNN). We systematically study different methods to solve this problem and compared them with the proposed method. Extensive experiments carried out on large image datasets demonstrate that our method outperforms the state-of-the-art both quantitatively and qualitatively. Finally, we have applied various loss functions in the proposed scheme and demonstrate that by using the mean absolute error as a loss function, we can obtain superior results in comparison to other cases.
StylePeople: A Generative Model of Fullbody Human Avatars

We propose a new type of full-body human avatars, which combines parametric mesh-based body model with a neural texture. We show that with the help of neural textures, such avatars can successfully model clothing and hair, which usually poses a problem for mesh-based approaches. We also show how these avatars can be created from multiple frames of a video using backpropagation. We then propose a generative model for such avatars that can be trained from datasets of images and videos of people. The generative model allows us to sample random avatars as well as to create dressed avatars of people from one or few images. The code for the project is available at saic-violet.github.io/style-people.
HedgeCut: Maintaining Randomised Trees for Low-Latency Machine Unlearning

Software systems that learn from user data with machine learning (ML) have become ubiquitous over the last years. Recent law such as the "General Data Protection Regulation" (GDPR) requires organisations that process personal data to delete user data upon request (enacting the "right to be forgotten"). However, this regulation does not only require the deletion of user data from databases, but also applies to ML models that have been learned from the stored data. We therefore argue that ML applications should offer users to unlearn their data from trained models in a timely manner. We explore how fast this unlearning can be done under the constraints imposed by real world deployments, and introduce the problem of low-latency machine unlearning: maintaining a deployed ML model in-place under the removal of a small fraction of training samples without retraining.
An Empirical Study on the Usage of BERT Models for Code Completion

Code completion is one of the main features of modern Integrated Development Environments (IDEs). Its objective is to speed up code writing by predicting the next code token(s) the developer is likely to write. Research in this area has substantially bolstered the predictive performance of these techniques. However, the support to developers is still limited to the prediction of the next few tokens to type. In this work, we take a step further in this direction by presenting a large-scale empirical study aimed at exploring the capabilities of state-of-the-art deep learning (DL) models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). To this aim, we train and test several adapted variants of the recently proposed RoBERTa model, and evaluate its predictions from several perspectives, including: (i) metrics usually adopted when assessing DL generative models (i.e., BLEU score and Levenshtein distance); (ii) the percentage of perfect predictions (i.e., the predicted code snippets that match those written by developers); and (iii) the "semantic" equivalence of the generated code as compared to the one written by developers. The achieved results show that BERT models represent a viable solution for code completion, with perfect predictions ranging from ~7%, obtained when asking the model to guess entire blocks, up to ~58%, reached in the simpler scenario of few tokens masked from the same code statement.
Ranking Tourist Attractions through Online Reviews: A Novel Method with Intuitionistic and Hesitant Fuzzy Information Based on Sentiment Analysis

Online tourist reviews are the real feeling of tourists after the journey, which have a strong reference value for potential tourists to make travel decisions. However, it is almost impossible for a potential tourist to look through the massive online reviews related to tourist attractions (TAs) so that he/she can make the most appropriate decision. To this end, this paper proposes a recommender system to rank the alternative TAs through online reviews based on aspect-level sentiment analysis and multi-criteria decision-making (MCDM) with intuitionistic and hesitant fuzzy information. In this methodology, the aspects that the experienced tourists concern are extracted from online reviews to construct a three-level evaluation system (including target layer, criteria layer and sub-criteria layer), which not only ensures the comprehensive evaluation of TAs as much as possible, but also reduces the complexity of the decision-making process. Then, the online reviews related to these sub-criteria are transformed into the corresponding intuitionistic and hesitant fuzzy performance scores through aspect-level sentiment analysis. Furthermore, in order to obtain the final ranking result that more in line with the expectations of the potential tourist, the preference information from the potential tourist and experienced tourists is integrated to determine the weights of criteria. Subsequently, the intuitionistic and hesitant fuzzy TOPSIS (IHF-TOPSIS) method is proposed to rank the alternative TAs. Finally, a case study is provided to verify the validity and applicability of the methodology.
Semi-Supervised Video Deraining with Dynamical Rain Generator

While deep learning (DL)-based video deraining methods have achieved significant successes in recent years, they still have two major drawbacks. Firstly, most of them are insufficient to model the characteristics of rain layers contained in rainy videos. In fact, the rain layers exhibit strong visual properties (e.g., direction, scale, and thickness) in spatial dimension and causal properties (e.g., velocity and acceleration) in temporal dimension, and thus can be modeled by the spatial-temporal process in statistics. Secondly, current DL-based methods rely heavily on the labeled training data, whose rain layers are synthetic, thus leading to a deviation from real data. Such a gap between synthetic and real data sets results in poor performance when applying them to real scenarios. To address these issues, this paper proposes a new semi-supervised video deraining method, in which a dynamical rain generator is employed to fit the rain layer for the sake of better depicting its intrinsic characteristics. Specifically, the dynamical generator consists of one emission model and one transition model to simultaneously encode the spatial appearance and temporal dynamics of rain streaks, respectively, both of which are parameterized by deep neural networks (DNNs). Furthermore, different prior formats are designed for the labeled synthetic and unlabeled real data so as to fully exploit their underlying common knowledge. Last but not least, we design a Monte Carlo-based EM algorithm to learn the model. Extensive experiments are conducted to verify the superiority of the proposed semi-supervised deraining model.
Correlation formulation for optimum tilt angle for maximizing the solar radiation on solar collector in the Western Himalayan region

Solar tracking devices are efficacious in maximizing solar irradiation collection. However, higher price makes these systems less frequently used. As an alternative, optimum tilt angle estimation has the advantage that it does not involve tracking cost expenditure. In this study, optimum tilt angles for different months have been computed for Hamirpur, Himachal Pradesh, India (L 31° 42′ N). Several mathematical models and statistical tools have been employed to forecast the monthly optimum tilt angles for the selected site. Different scenarios have been presented by considering five, four, three, and two annual adjustment models to increase the practical usage of the estimated optimum tilt angles. Additionally, cost-benefit analysis has also be performed on the PV panels. Based on the results, it has been concluded that model M − 11, with three annual adjustments, stipulates maximum benefits over the other models. This study can serve as fundamental guidance for setting up solar energy plants in this specified region with the highest efficiency.
Hybrid-Order Gated Graph Neural Network for Session-Based Recommendation

Considering sessions as directed subgraphs, graph neural networks (GNNs) are supposed to be capable of capturing the complex dependencies among items and suitable for session-based recommendation. However, deep GNNs suffer from the oversmoothing problem of making all nodes converge to the same value. In session-based recommendation, the subgraphs transformed by short sessions are usually simple, which cause worse oversmoothing problem. To apply GNNs to session-based recommendation sufficiently, in this article, we propose a hybrid-order gated GNN (HGNN) on account of the oversmoothing problem. The proposed HGNN model is based on the hybrid-order propagation, which avoids insignificant patterns and captures complex dependencies in propagation. What's more, the attention mechanism is utilized to learn different weights of orders in propagation. Then, HGNN is applied to session-based recommendation, which results in a new method called SR-HGNN. Experimental results show that SR-HGNN outperforms the state-of-the-art session-based recommendation methods and eases the oversmoothing problem.