Reviewerly/exHarmony huggingface dataset:
ConViT: improving vision transformers with soft convolutional inductive biases*

Convolutional architectures have proven extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision Transformers (ViTs) rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a ``soft" convolutional inductive bias. We initialise the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutional-like ViT architecture, ConViT, outperforms the DeiT on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analysing how it is escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at https://github.com/facebookresearch/convit.
End-to-End Semi-Supervised Object Detection with Soft Teacher

This paper presents an end-to-end semi-supervised object detection approach, in contrast to previous more complex multi-stage methods. The end-to-end training gradually improves pseudo label qualities during the curriculum, and the more and more accurate pseudo labels in turn benefit object detection training. We also propose two simple yet effective techniques within this framework: a soft teacher mechanism where the classification loss of each unlabeled bounding box is weighed by the classification score produced by the teacher network; a box jittering approach to select reliable pseudo boxes for the learning of box regression. On the COCO benchmark, the proposed approach outperforms previous methods by a large margin under various labeling ratios, i.e. 1%, 5% and 10%. Moreover, our approach proves to perform also well when the amount of labeled data is relatively large. For example, it can improve a 40.9 mAP baseline detector trained using the full COCO training set by +3.6 mAP, reaching 44.5 mAP, by leveraging the 123K unlabeled images of COCO. On the state-of-the-art Swin Transformer based object detector (58.9 mAP on test-dev), it can still significantly improve the detection accuracy by +1.5 mAP, reaching 60.4 mAP, and improve the instance segmentation accuracy by +1.2 mAP, reaching 52.4 mAP. Further incorporating with the Object365 pre-trained model, the detection accuracy reaches 61.3 mAP and the instance segmentation accuracy reaches 53.0 mAP, pushing the new state-of-the-art. The code and models will be made publicly available at https://github.com/microsoft/SoftTeacher.
Photonic matrix multiplication lights up photonic accelerator and beyond

Matrix computation, as a fundamental building block of information processing in science and technology, contributes most of the computational overheads in modern signal processing and artificial intelligence algorithms. Photonic accelerators are designed to accelerate specific categories of computing in the optical domain, especially matrix multiplication, to address the growing demand for computing resources and capacity. Photonic matrix multiplication has much potential to expand the domain of telecommunication, and artificial intelligence benefiting from its superior performance. Recent research in photonic matrix multiplication has flourished and may provide opportunities to develop applications that are unachievable at present by conventional electronic processors. In this review, we first introduce the methods of photonic matrix multiplication, mainly including the plane light conversion method, Mach-Zehnder interferometer method and wavelength division multiplexing method. We also summarize the developmental milestones of photonic matrix multiplication and the related applications. Then, we review their detailed advances in applications to optical signal processing and artificial neural networks in recent years. Finally, we comment on the challenges and perspectives of photonic matrix multiplication and photonic acceleration.
RuleMatrix: Visualizing and Understanding Classifiers with Rules

With the growing adoption of machine learning techniques, there is a surge of research interest towards making machine learning systems more transparent and interpretable. Various visualizations have been developed to help model developers understand, diagnose, and refine machine learning models. However, a large number of potential but neglected users are the domain experts with little knowledge of machine learning but are expected to work with machine learning systems. In this paper, we present an interactive visualization technique to help users with little expertise in machine learning to understand, explore and validate predictive models. By viewing the model as a black box, we extract a standardized rule-based knowledge representation from its input-output behavior. Then, we design RuleMatrix, a matrix-based visualization of rules to help users navigate and verify the rules and the black-box model. We evaluate the effectiveness of RuleMatrix via two use cases and a usability study.
Trading volume and the predictability of return and volatility in the cryptocurrency market

We extend our limited understanding on the Granger causality from trading volume to the returns and volatility in the cryptocurrency market via a copula-quantile causality approach. Using daily data of seven leading cryptocurrencies (Bitcoin, Ripple, Ethereum, Litecoin, Nem, Dash, and Stellar), results show that trading volume Granger causes extreme negative and positive returns of all cryptocurrencies under study. However, volume Granger causes return volatility for only three cryptocurrencies (Litecoin, NEM, and Dash) when the volatility is low. However, this latter result only holds when squared returns are used as a proxy of volatility and not when GARCH volatility is employed.
One-Step Multi-View Spectral Clustering

Previous multi-view spectral clustering methods are a two-step strategy, which first learns a fixed common representation (or common affinity matrix) of all the views from original data and then conducts k-means clustering on the resulting common affinity matrix. The two-step strategy is not able to output reasonable clustering performance since the goal of the first step (i.e., the common affinity matrix learning) is not designed for achieving the optimal clustering result. Moreover, the two-step strategy learns the common affinity matrix from original data, which often contain noise and redundancy to influence the quality of the common affinity matrix. To address these issues, in this paper, we design a novel One-step Multi-view Spectral Clustering (OMSC) method to output the common affinity matrix as the final clustering result. In the proposed method, the goal of the common affinity matrix learning is designed to achieving optimal clustering result and the common affinity matrix is learned from low-dimensional data where the noise and redundancy of original high-dimensional data have been removed. We further propose an iterative optimization method to fast solve the proposed objective function. Experimental results on both synthetic datasets and public datasets validated the effectiveness of our proposed method, comparing to the state-of-the-art methods for multi-view clustering.
Blockchain for decentralized transactive energy management system in networked microgrids

The proliferation of distributed energy resources is reshaping the landscape of power distribution systems, including a network of autonomous microgrids. Networked microgrids transact energy for managing the efficiency, reliability, resilience, security, and sustainability of electric power services. This article offers a vision and analyzes a scheme developed for networked microgrids that utilizes blockchain technologies to optimize the financial and physical operations of power distribution systems. Blockchain provides a powerful and trustworthy path for launching distributed data storage and management, the article explores the possibility of customizing blockchain technologies to meet socioeconomic requirements of transactive energy management at the power distribution level. Then, a set of interoperable blockchains embedded with self-enforcing smart contracts is proposed to manage energy and financial flows among transacting microgrids in a credible manner. The article presents additional smart contract measures for securing optimal energy transactions between networked microgrids and the local distribution grid. It is concluded that blockchain technologies embedded in transactive energy will play a significant role in the evolution of traditional power distribution systems to active distribution networks.
On cloud security requirements, threats, vulnerabilities and countermeasures: A survey

The world is witnessing a phenomenal growth in the cloud enabled services and is expected to grow further with the improved technological innovations. However, the associated security and privacy challenges inhibit its widespread adoption, and therefore require further exploration. Researchers from academia, industry, and standards organizations have provided potential solutions to these challenges in the previously published studies. The narrative review presented in this survey, however, provides an integrationist end-to-end mapping of cloud security requirements, identified threats, known vulnerabilities, and recommended countermeasures, which seems to be not presented before at one place. Additionally, this study contributes towards identifying a unified taxonomy for security requirements, threats, vulnerabilities and countermeasures to carry out the proposed end-to-end mapping. Further, it highlights security challenges in other related areas like trust based security models, cloud-enabled applications of Big Data, Internet of Things (IoT), Software Defined Network (SDN) and Network Function Virtualization (NFV).
Self-Supervised Representation Learning by Rotation Feature Decoupling

We introduce a self-supervised learning method that focuses on beneficial properties of representation and their abilities in generalizing to real-world tasks. The method incorporates rotation invariance into the feature learning framework, one of many good and well-studied properties of visual representation, which is rarely appreciated or exploited by previous deep convolutional neural network based self-supervised representation learning methods. Specifically, our model learns a split representation that contains both rotation related and unrelated parts. We train neural networks by jointly predicting image rotations and discriminating individual instances. In particular, our model decouples the rotation discrimination from instance discrimination, which allows us to improve the rotation prediction by mitigating the influence of rotation label noise, as well as discriminate instances without regard to image rotations. The resulting feature has a better generalization ability for more various tasks. Experimental results show that our model outperforms current state-of-the-art methods on standard self-supervised feature learning benchmarks.
A Collaborative Session-based Recommendation Approach with Parallel Memory Modules

Session-based recommendation is the task of predicting the next item to recommend when the only available information consists of anonymous behavior sequences. Previous methods for session-based recommendation focus mostly on the current session, ignoring collaborative information in so-called neighborhood sessions, sessions that have been generated previously by other users and reflect similar user intents as the current session. We hypothesize that the collaborative information contained in such neighborhood sessions may help to improve recommendation performance for the current session.
Supervised Classification Algorithms in Machine Learning: A Survey and Review

Machine learning is currently one of the hottest topics that enable machines to learn from data and build predictions without being explicitly programmed for that task, automatically without human involvement. Supervised learning is one of two broad branches of machine learning that makes the model enable to predict future outcomes after they are trained based on past data where we use input/output pairs or the labeled data to train the model with the goal to produce a function that is approximated enough to be able to predict outputs for new inputs when introduced to them. Supervised learning problems can be grouped into regression problems and classification problems. A regression problem is when outputs are continuous whereas a classification problem is when outputs are categorical. This paper tries to compare different types of classification algorithms precisely widely used ones on the basis of some basic conceptions though it is obvious that a complete and comprehensive review and survey of all the supervised learning classification algorithms possibly cannot be accomplished by a single paper, but the references cited in this paper hopefully cover the significant theoretical issues and our survey has been kept limited to the widely used algorithms because the field is highly growing and not possible to cover all the algorithms in a single paper. One more point to be mentioned here that any study of complex procedure like neural networks has not been included as it has been tried to keep the content as much simple as possible.
Robustness via Curvature Regularization, and Vice Versa

State-of-the-art classifiers have been shown to be largely vulnerable to adversarial perturbations. One of the most effective strategies to improve robustness is adversarial training. In this paper, we investigate the effect of adversarial training on the geometry of the classification landscape and decision boundaries. We show in particular that adversarial training leads to a significant decrease in the curvature of the loss surface with respect to inputs, leading to a drastically more "linear" behaviour of the network. Using a locally quadratic approximation, we provide theoretical evidence on the existence of a strong relation between large robustness and small curvature. To further show the importance of reduced curvature for improving the robustness, we propose a new regularizer that directly minimizes curvature of the loss surface, and leads to adversarial robustness that is on par with adversarial training. Besides being a more efficient and principled alternative to adversarial training, the proposed regularizer confirms our claims on the importance of exhibiting quasi-linear behavior in the vicinity of data points in order to achieve robustness.
Image Restoration by Iterative Denoising and Backward Projections

Inverse problems appear in many applications, such as image deblurring and inpainting. The common approach to address them is to design a specific algorithm for each problem. The Plug-and-Play (P&P) framework, which has been recently introduced, allows solving general inverse problems by leveraging the impressive capabilities of existing denoising algorithms. While this fresh strategy has found many applications, a burdensome parameter tuning is often required in order to obtain high-quality results. In this paper, we propose an alternative method for solving inverse problems using off-the-shelf denoisers, which requires less parameter tuning. First, we transform a typical cost function, composed of fidelity and prior terms, into a closely related, novel optimization problem. Then, we propose an efficient minimization scheme with a P&P property, i.e., the prior term is handled solely by a denoising operation. Finally, we present an automatic tuning mechanism to set the method's parameters. We provide a theoretical analysis of the method and empirically demonstrate its competitiveness with task-specific techniques and the P&P approach for image inpainting and deblurring.
Gaussian mutational chaotic fruit fly-built optimization and feature selection

To cope with the potential shortcomings of classical fruit fly optimization algorithm (FOA), a new version of FOA with Gaussian mutation operator and the chaotic local search strategy (MCFOA) is proposed in this research. First, the Gaussian mutation operator is introduced into the basic FOA to avoid premature convergence and improve the exploitative tendencies in the algorithm (MFOA). Then, chaotic local search method is adopted for enhancing the local searching ability of the swarm of agents (CFOA). To substantiate the efficiency of three proposed methods, a comprehensive comparison has been completed using 23 benchmark functions with different characteristics. The best version of FOA among them is the MCFOA, which is extensively compared with the notable swarm-intelligence algorithms like bat algorithm (BA), particle swarm optimization algorithm (PSO), and several advanced FOA-based methods such as chaotic FOA (CIFOA), improved FOA (IFOA), multi-swarm FOA (swarm_MFOA) and differential evolution based FOA (DFOA). Numerical results show that two embedded strategies will effectively boost the performance of FOA for optimization tasks. In addition, MCFOA is also applied to feature selection problems. The results also prove that MCFOA can obtain the optimal classification accuracy.
SparTen

Convolutional neural networks (CNNs) are emerging as powerful tools for image processing. Recent machine learning work has reduced CNNs' compute and data volumes by exploiting the naturally-occurring and actively-transformed zeros in the feature maps and filters. While previous semi-sparse architectures exploit one-sided sparsity either in the feature maps or the filters, but not both, a recent fully-sparse architecture, called Sparse CNN (SCNN), exploits two-sided sparsity to improve performance and energy over dense architectures. However, sparse vector-vector dot product, a key primitive in sparse CNNs, would be inefficient using the representation adopted by SCNN. The dot product requires finding and accessing non-zero elements in matching positions in the two sparse vectors -- an inner join using the position as the key with a single value field. SCNN avoids the inner join by performing a Cartesian product capturing the relevant multiplications. However, SCNN's approach incurs several considerable overheads and is not applicable to non-unit-stride convolutions. Further, exploiting reuse in sparse CNNs fundamentally causes systematic load imbalance not addressed by SCNN. We propose SparTen which achieves efficient inner join by providing support for native two-sided sparse execution and memory storage. To tackle load imbalance, SparTen employs a software scheme, called greedy balancing, which groups filters by density via two variants, a software-only one which uses whole-filter density and a software-hardware hybrid which uses finer-grain density. Our simulations show that, on average, SparTen performs 4.7x, 1.8x, and 3x better than a dense architecture, one-sided sparse architecture, and SCNN, respectively. An FPGA implementation shows that SparTen performs 4.3x and 1.9x better than a dense architecture and a one-sided sparse architecture, respectively.
ThunderNet: Towards Real-Time Generic Object Detection on Mobile Devices

Real-time generic object detection on mobile platforms is a crucial but challenging computer vision task. Prior lightweight CNN-based detectors are inclined to use one-stage pipeline. In this paper, we investigate the effectiveness of two-stage detectors in real-time generic detection and propose a lightweight two-stage detector named ThunderNet. In the backbone part, we analyze the drawbacks in previous lightweight backbones and present a lightweight backbone designed for object detection. In the detection part, we exploit an extremely efficient RPN and detection head design. To generate more discriminative feature representation, we design two efficient architecture blocks, Context Enhancement Module and Spatial Attention Module. At last, we investigate the balance between the input resolution, the backbone, and the detection head. Benefit from the highly efficient backbone and detection part design, ThunderNet surpasses previous lightweight one-stage detectors with only 40% of the computational cost on PASCAL VOC and COCO benchmarks. Without bells and whistles, ThunderNet runs at 24.1 fps on an ARM-based device with 19.2 AP on COCO. To the best of our knowledge, this is the first real-time detector reported on ARM platforms. Code will be released for paper reproduction.
[DL] A Survey of FPGA-based Neural Network Inference Accelerators

Recent research on neural networks has shown a significant advantage in machine learning over traditional algorithms based on handcrafted features and models. Neural networks are now widely adopted in regions like image, speech, and video recognition. But the high computation and storage complexity of neural network inference poses great difficulty on its application. It is difficult for CPU platforms to offer enough computation capacity. GPU platforms are the first choice for neural network processes because of its high computation capacity and easy-to-use development frameworks. However, FPGA-based neural network inference accelerator is becoming a research topic. With specifically designed hardware, FPGA is the next possible solution to surpass GPU in speed and energy efficiency. Various FPGA-based accelerator designs have been proposed with software and hardware optimization techniques to achieve high speed and energy efficiency. In this article, we give an overview of previous work on neural network inference accelerators based on FPGA and summarize the main techniques used. An investigation from software to hardware, from circuit level to system level is carried out to complete analysis of FPGA-based neural network inference accelerator design and serves as a guide to future work.
Blockchain-Based On-Demand Computing Resource Trading in IoV-Assisted Smart City

In a smart city, Mobile Edge Computing (MEC) are generally deployed in static fashion in base stations (BSs). While moving vehicles with advanced on-board equipment can be regarded as dynamic computing resource transporters ignoring geographical limitations. Thus Internet of Vehicle (IoV) could assist the smart city to achieve flexible computing resource demand response (DR) via paid sharing the idle vehicle computing resources. Motivated by this, we propose a Peer-to-Peer (P2P) computing resource trading system to balance computing resource spatio-temporal dynamic demands in IoV-assisted smart city. On one hand, to guarantee transaction security and privacy-preserving in our system, we employ a consortium blockchain approach and demonstrate the process of secure computing resource trading without involving a centralized trusted third-party. On the other hand, to encourage individual smart vehicles to participate in our system, we construct a two-stage Stackelberg game jointly optimizing the utilities of buyers and sellers. And we also derive the optimal computing pricing and trading amount strategies in this proposed game. Finally, security analysis shows the security performance of our system and numerical simulations show that our strategies can encourage the collaboration between the buyer and smart vehicles.