Reviewerly/exHarmony huggingface dataset:
Non-uniform Step Size Quantization for Accurate Post-training Quantization

Quantization is a very effective optimization technique to reduce hardware cost and memory footprint of deep neural network (DNN) accelerators. In particular, post-training quantization (PTQ) is often preferred as it does not require a full dataset or costly retraining. However, performance of PTQ lags significantly behind that of quantization-aware training especially for low-precision networks ( $$\le $$ 4-bit). In this paper we propose a novel PTQ scheme (Code will be publicly available at https://github.com/sogh5/SubsetQ ) to bridge the gap, with minimal impact on hardware cost. The main idea of our scheme is to increase arithmetic precision while retaining the same representational precision. The excess arithmetic precision enables us to better match the input data distribution while also presenting a new optimization problem, to which we propose a novel search-based solution. Our scheme is based on logarithmic-scale quantization, which can help reduce hardware cost through the use of shifters instead of multipliers. Our evaluation results using various DNN models on challenging computer vision tasks (image classification, object detection, semantic segmentation) show superior accuracy compared with the state-of-the-art PTQ methods at various low-bit precisions.
Abstracting Sketches Through Simple Primitives

Humans show high-level of abstraction capabilities in games that require quickly communicating object information. They decompose the message content into multiple parts and communicate them in an interpretable protocol. Toward equipping machines with such capabilities, we propose the Primitive-based Sketch Abstraction task where the goal is to represent sketches using a fixed set of drawing primitives under the influence of a budget. To solve this task, our Primitive-Matching Network (PMN), learns interpretable abstractions of a sketch in a self supervised manner. Specifically, PMN maps each stroke of a sketch to its most similar primitive in a given set, predicting an affine transformation that aligns the selected primitive to the target stroke. We learn this stroke-to-primitive mapping end-to-end with a distance-transform loss that is minimal when the original sketch is precisely reconstructed with the predicted primitives. Our PMN abstraction empirically achieves the highest performance on sketch recognition and sketch-based image retrieval given a communication budget, while at the same time being highly interpretable. This opens up new possibilities for sketch analysis, such as comparing sketches by extracting the most relevant primitives that define an object category. Code is available at https://github.com/ExplainableML/sketch-primitives .
Unsupervised Neural Machine Translation for English to Kannada Using Pre-Trained Language Model

With the advent of Neural Networks, Neural Machine Translation (NMT) has been a booming area of research in Natural Language Processing. Indic languages possess less significant parallel and monolingual corpus than English and other European languages. Even in Indic languages, many Indo-Aryan and Dravidian languages fall into low-resource languages, and techniques like Back Translation, Unsupervised Neural Machine Translation (UNMT), and Transfer Learning help achieve better translation accuracy. This paper uses the UNMT approach with a pre-trained Cross-Lingual Language Model (XLM) for the English to Kannada language pair. Our proposed methodology has achieved a 0.61 BLEU score in English to Kannada and a 0.32 BLEU score in Kannada to English translation.
What makes you, you? Analyzing Recognition by Swapping Face Parts

Deep learning advanced face recognition to an unprecedented accuracy. However, understanding how local parts of the face affect the overall recognition performance is still mostly unclear. Among others, face swap has been experimented to this end, but just for the entire face. In this paper, we propose to swap facial parts as a way to disentangle the recognition relevance of different face parts, like eyes, nose and mouth. In our method, swapping parts from a source face to a target one is performed by fitting a 3D prior, which establishes dense pixels correspondence between parts, while also handling pose differences. Seamless cloning is then used to obtain smooth transitions between the mapped source regions and the shape and skin tone of the target face. We devised an experimental protocol that allowed us to draw some preliminary conclusions when the swapped images are classified by deep networks, indicating a prominence of the eyes and eyebrows region. Code available at https://github.com/clferrari/FacePartsSwap
Kernelized Transfer Joint Matching for Unsupervised Domain Adaptation

Transfer learning is an emerging technique through which the machine can learn a new task from the previous experience of another related task to solve the problem of insufficient labelled data. Existing work has explored two transfer learning strategies: feature matching and instance reweighting independently or jointly. However, when we deal with nonlinear training and test data, these strategies may fail. Therefore, to handle this situation, in this paper, we propose a novel kernelized transfer joint matching (KTJM) approach. More specifically, KTJM aims to minimize the distribution gap between the source and the target domains by jointly matching the features and reweighting the instances across domains with Laplacian regularization in a kernelized unified framework. Basically, KTJM constructs a new invariant feature space for both irrelevant instances and distribution gap in an RKHS using d-dimensional embeddings extracted by the kernelized PCA. Extensive experimental results have verified that the proposed kernelized framework outperforms several state-of-the-art transfer learning or domain adaptation methods on the Office-Caltech10 and PIE face data sets. Particularly, KTJM achieved an average accuracy of 90.2% and 79.342% for all classification tasks of Office-Caltech10 data set using Decaf features and PIE face data set, respectively.
Black-Box Adversarial Attacks on Deep Neural Networks: A Survey

Deep neural networks are capable of performing many challenging tasks, such as image classification, speech recognition and game playing. However, recent research shows that deep neural networks can be tricked by adversarial attacks, which craft adversarial examples by adding subtle perturbations to the clean examples. Adversarial attacks can be broadly divided into white-box attacks and black-box attacks. This paper presents a comprehensive survey on black-box attacks. The black-box attacks are divided into three categories: transfer-based attacks, score-based attacks and decision-based attacks. We also introduce some existing defenses against black-box attacks.
Hydrophobicity Classification of Composite Insulators Based on Faster R-CNN Object Detection Algorithm

Composite insulators are used in a large number of high-voltage transmission systems in China, but because of environmental factors and other effects, the surface of composite insulators will gradually undergo insulation degradation. The water repellent property of composite insulator surface is one of the indicators to reflect its insulation condition. The traditional method of judging the hydrophobicity level of composite insulator surface by manual work is subjective and not very efficient, while the digital image processing-based method of evaluating the hydrophobicity of composite insulator reduces the workload of the evaluation process by using computer algorithms, but the proposed characteristic parameters are still subjective. Therefore, this paper proposes a composite insulator hydrophobicity classification model based on Faster R-CNN target detection algorithm, which can automatically complete the evaluation of composite insulator water repellency, and the accuracy of this model is close to 99% with the allowed error of ±1
BITKOMO: Combining Sampling and Optimization for Fast Convergence in Optimal Motion Planning

Optimal sampling based motion planning and trajectory optimization are two competing frameworks to generate optimal motion plans. Both frameworks have complementary properties: Sampling based planners are typically slow to converge, but provide optimality guarantees. Trajectory optimizers, however, are typically fast to converge, but do not provide global optimality guarantees in nonconvex problems, e.g. scenarios with obstacles. To achieve the best of both worlds, we introduce a new planner, BITKOMO, which integrates the asymptotically optimal Batch Informed Trees (BIT*) planner with the K-Order Markov Optimization (KOMO) trajectory optimization framework. Our planner is anytime and maintains the same asymptotic optimality guarantees provided by BIT*, while also exploiting the fast convergence of the KOMO trajectory optimizer. We experimentally evaluate our planner on manipulation scenarios that involve high dimensional configuration spaces, with up to two 7-DoF manipulators, obstacles and narrow passages. BITKOMO performs better than KOMO by succeeding even when KOMO fails, and it outperforms BIT* in terms of convergence to the optimal solution.
Scene Text Recognition with Single-Point Decoding Network

In recent years, attention-based scene text recognition methods have been very popular and attracted the interest of many researchers. Attention-based methods can adaptively focus attention on a small area or even single point during decoding, in which the attention matrix is nearly one-hot distribution. Furthermore, the whole feature maps will be weighted and summed by all attention matrices during inference, causing huge redundant computations. In this paper, we propose an efficient attention-free Single-Point Decoding Network (dubbed SPDN) for scene text recognition, which can replace the traditional attention-based decoding network. Specifically, we propose Single-Point Sampling Module (SPSM) to efficiently sample one key point on the feature map for decoding one character. In this way, our method can not only precisely locate the key point of each character but also remove redundant computations. Based on SPSM, we design an efficient and novel single-point decoding network to replace the attention-based decoding network. Extensive experiments on publicly available benchmarks verify that our SPDN can greatly improve decoding efficiency without sacrificing performance.
Deep Learning techniques for Object Detection in Images: Exploratory Study

The aim of computer vision is to visualize the data available in images or videos. Visualizing such objects by machine is known as object detection in images. It is one of the most recent research areas. It is used to detect and recognize various objects present in an image. Object detection can be used in detecting tumors in the medical field, animal detection in agriculture, detecting whether a person is carrying any weapon or not, in e-commerce sites, image-based search engines, self-driving cars, and many more. Convolutional Neural Network (ConvNet or CNN) is a dominant technology that is used in detecting objects because it makes the process fast. The area is quite popular among researchers as well as industry person. There are various techniques used for object detection like CNN, R-CNN, Fast RCNN, and Faster RCNN. In this paper, a comparative study of all these techniques is done to show which one is better than other.
Self-supervised Spatial Reasoning on Multi-View Line Drawings

Spatial reasoning on multi-view line drawings by state-of-the-art supervised deep networks is recently shown with puzzling low performances on the SPARE3D dataset [14]. Based on the fact that self-supervised learning is helpful when a large number of data are available, we propose two self-supervised learning approaches to improve the baseline performance for view consistency reasoning and camera pose reasoning tasks on the SPARE3D dataset. For the first task, we use a self-supervised binary classification network to contrast the line drawing differences between various views of any two similar 3D objects, enabling the trained networks to effectively learn detail-sensitive yet view-invariant line drawing representations of 3D objects. For the second type of task, we propose a self-supervised multi-class classification framework to train a model to select the correct corresponding view from which a line drawing is rendered. Our method is even helpful for the downstream tasks with unseen camera poses. Experiments show that our method could significantly increase the baseline performance in SPARE3D, while some popular self-supervised learning methods cannot.
A Non-Local Attention Feature Fusion Network for Multiscale Object Detection

Feature pyramid networks (FPN) provide typical architectures for building learning networks with advanced semantic features, which are essential for object recognition at different scales. However, FPN have severe shortcomings in the feature extraction and fusion stages, such as making the extracted features lack of contextual and deep semantic information. In this work, we propose a non-local channel and spatial attention feature pyramid network (NCS-FPN) to improve multi-scale learning. In the feature extraction process, contextual semantic information from different scales is collected through non-local attention networks. In the feature fusion phase, deeper feature information from both spatial and context-aware sources is aggregated to enhance multi-scale feature extraction. Extensive experiments are carried out on two public datasets, MS COCO and PASCAL VOC, are carried out and the results demonstrate that NCS-FPN achieves better performance than several SOTA methods in most cases.
Attention-Based Relation Prediction of Knowledge Graph by Incorporating Graph and Context Features

As the vital resource for various applications like question answering and recommendation system, knowledge graph (KG) often suffers from incompleteness. The task of relation prediction in KG aims to infer the relations between entities, which depend on the structure of the query-specific subgraph but also the neighborhood as context. In this paper, we propose an attention-based joint model for relation prediction by incorporating the graph and context features of entities and relations. First, we extract the subgraph and entity context from KG, and adopt the attention mechanism to capture the most relevant graph and context features, which we leverage to generate the enhanced representations. Then, we give the joint loss function to guarantee the unified representation including the relevant graph and context features simultaneously. Finally, we fulfill model training for relation prediction. Experimental results on real-world datasets demonstrate that our proposed approach outperforms the state-of-the-art competitors.
Avenger: Punishing the Cross-Core Last-Level Cache Attacker and Not the Victim by Isolating the Attacker

On a multi-core system, the shared last-level cache(LLC) is vulnerable to various kinds of cross-core contention-based attacks. LLC randomization and LLC partitioning are two promising mitigation strategies that mitigate these attacks. LLC-randomization techniques make an attacker’s life difficult in mounting contention-based attacks but do not entirely mitigate them. Randomized caches are also ineffective in preventing occupancy-based attacks. In contrast, state-of-the-art LLC partitioning techniques mitigate all possible LLC contention-based attacks by allocating isolated LLC regions to different processes or security domains. However, restricting processes to isolated LLC region(s) affects overall LLC utilization and incurs performance overhead (as high as 72%) and memory subsystem energy overhead (as high as 89%); effectively providing security guarantee at the cost of performance and energy.One of the primary reasons for this trend is the fundamental design choice that drives all the state-of-the-art secure LLC partitioning techniques, which isolate all the applications and all the time at the LLC. We revisit this choice and argue that we need to isolate only the attacker process and not all the processes. To isolate the attacker, we propose Avenger, a mitigation technique that uses a state-of-the-art LLC contention attack detector and isolates only the attacker. The detector is flexible and can be trained as per the security requirements of any organization or cloud provider. Experimental results on a 16-core simulated system with one attacker and 15 victims show that Avenger outperforms three state-of-the-art secure LLC partitioning techniques in performance and energy overhead without affecting security. Overall, Avenger provides a robust security guarantee against all contention-based cross-core LLC attacks with 1% average performance overhead in contrast to an average performance overhead of more than 17% with the state-of-the-art secure LLC partitioning techniques.
Building an IPFS and Blockchain-Based Decentralized Storage Model for Medical Imaging

Currently, sharing and access of medical imaging is a significant element of present healthcare systems, but the existing infrastructure of medical image sharing depends on third-party approval. In this chapter, the authors have proposed a framework in order to provide a decentralized storage model for medical image sharing through IPFS and blockchain technology that remove the hurdle of third-party dependency. In the proposed model, the authors are sharing the imaging and communications in medicine (DICOM) medical images, which consist of various information related to disease, and hence, the framework can be utilized in the real-time application of the healthcare system. Moreover, the framework maintains the feature of immutability, privacy, and availability of information owing to the blockchain-based decentralized storage model. Furthermore, the authors have also discussed how the information can be accessed by the peers in the blockchain network with the help of consensus. To implement the framework, they have used the python ask and anaconda python.
An Energy-Efficient Double Ratchet Cryptographic Processor With Backward Secrecy for IoT Devices

This work presents the first cryptographic processor that supports the double ratchet protocol with backward secrecy for the Internet-of-Things (IoT) devices. A precomputation-based constant modular divider is used to reduce the area by 39.5% and energy consumption by 18.8%. A hash-based key derivative function (HKDF) module is proposed to reduce the energy consumption of the length selector by 89.8% and the energy consumption of the module by 35% by leveraging the characteristic of the input. A <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${\mathrm{ GF}}(2^{4})^{2}$ </tex-math></inline-formula> -based S-box is used to reduce the area of S-box by 21.2%. A 1-byte S-box is shared for key generation and text encryption/decryption to reduce the area by 46.8%. Fabricated in a 40-nm CMOS technology, the chip integrates 227k gates in 1.03 mm 2 and dissipates 1.18 mW at 16 MHz from a 0.56-V supply. The chip achieves a 211 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$734\times $ </tex-math></inline-formula> lower energy consumption than the CPU solution. Compared with the state-of-the-art end-to-end protocol cryptographic processor, this work achieves an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$18.5\times $ </tex-math></inline-formula> higher energy efficiency for secure hash algorithm (SHA), <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3\times $ </tex-math></inline-formula> higher energy efficiency for advanced encryption standard (AES), and a 41% less energy for protocol establishment, with 10% smaller area.
Structural Rules and Algebraic Properties of Intersection Types

In this paper we define several notions of term expansion, used to define terms with less sharing, but with the same computational properties of terms typable in an intersection type system. Expansion relates terms typed by associative, commutative and idempotent intersections with terms typed in the Curry type system and the relevant type system; terms typed by non-idempotent intersections with terms typed in the affine and linear type systems; and terms typed by non-idempotent and non-commutative intersections with terms typed in an ordered type system.
Deep sparse representation via deep dictionary learning for reinforcement learning

Reinforcement Learning (RL) currently has been widely applied for various fields of decision-making, which relies on the function approximation for tackling complicated tasks. However, most RL models including Deep Reinforcement Learning (DRL) suffer from the catastrophic interference, resulting in poor control performance. Recently, the sparse representation has been proven effective for avoiding the interference and is easy to train, but shallow sparse representations have limited representation capability. In this paper, we proposed to employ the deep dictionary learning (DDL) model for reinforcement learning as a joint optimization problem over the deep sparse representation and the function approximation, which considers a multi-layer dictionary for obtaining a deeper latent sparse representation with better data representation capability. Firstly, to address the difficulties in optimization of DDL, we employ the decomposition method that we can decompose the multi-layer problem into a set of single-layer subproblems. Secondly, to solve the joint optimization problem efficiently, we apply the alternating update scheme for updating the sparse representation, multi-layer dictionary, and weight for value prediction alternately. Thirdly, we can directly obtain the closed-form solution of the sparse representation by the Proximal Operator (PO), multi-layer dictionary and weight are trained by the gradient descent method. Finally, the agent utilizes the pre-trained multi-layer dictionary and weight for RL control, and the weight is updated dynamically at each step. The experimental results on two benchmark environments demonstrate that the performance of the RL control training based on the DDL model can be significantly improved.
Uncertainty-based Visual Question Answering: Estimating Semantic Inconsistency between Image and Knowledge Base

Knowledge-based visual question answering (KVQA) task aims to answer questions that require additional external knowledge as well as an understanding of images and questions. Recent studies on KVQA inject an external knowledge in a multi-modal form, and as more knowledge is used, irrelevant information may be added and can confuse the question answering. In order to properly use the knowledge, this study proposes the following: 1) we introduce a novel semantic inconsistency measure computed from caption uncertainty and semantic similarity; 2) we suggest a new external knowledge assimilation method based on the semantic inconsistency measure and apply it to integrate explicit knowledge and implicit knowledge for KVQA; 3) the proposed method is evaluated with the OK-VQA dataset and achieves the state-of-the-art performance.