Reviewerly/exHarmony huggingface dataset:
Device-independent quantum secure direct communication against collective attacks

"Device-independent" not only represents a relaxation of the security assumptions about the internal working of the quantum devices, but also can enhance the security of the quantum communication. In the paper, we put forward the first device-independent quantum secure direct communication (DI-QSDC) protocol and analyze its security and communication efficiency against collective attacks. Under practical noisy quantum channel condition, the photon transmission loss and photon state decoherence would reduce DI-QSDC's communication quality and threaten its absolute security. For solving the photon transmission loss and decoherence problems, we adopt noiseless linear amplification (NLA) protocol and entanglement purification protocol (EPP) to modify the DI-QSDC protocol. With the help of the NLA and EPP, we can guarantee DI-QSDC's absolute security and effectively improve its communication quality.
Blockchain for Internet of Energy management: Review, solutions, and challenges

After smart grid, Internet of Energy (IoE) has emerged as a popular technology in the energy sector by integrating different forms of energy. IoE uses Internet to collect, organize, optimize and manage the networks energy information from different edge devices in order to develop a distributed smart energy infrastructure. Sensors and communication technologies are used to collect data and to predict demand and supply by consumers and suppliers respectively. However, with the development of renewable energy resources, Electric Vehicles (EVs), smart grid and Vehicle-to-grid (V2G) technology, the existing energy sector started shifting towards distributed and decentralized solutions. Moreover, the security and privacy issues because of centralization is another major concern for IoE technology. In this context, Blockchain technology with the features of automation, immutability, public ledger facility, irreversibility, decentralization, consensus and security has been adopted in the literature for solving the prevailing problems of centralized IoE architecture. By leveraging smart contracts, blockchain technology enables automated data exchange, complex energy transactions, demand response management and Peer-to-Peer (P2P) energy trading etc. Blockchain will play vital role in the evolution of the IoE market as distributed renewable resources and smart grid network are being deployed and used. We discuss the potential and applications of blockchain in the IoE field. This article is build on the literature research and it provides insight to the end-user regarding the future IoE scenario in the context of blockchain technology. Lastly this article discusses the different consensus algorithm for IoE technology.
A novel decentralized platform for peer-to-peer energy trading market with blockchain technology

Peer-to-Peer (P2P) energy trading, which allows energy consumers/producers to directly trade with each other, is one of the new paradigms driven by the decarbonization, decentralization, and digitalization of the energy supply chain. Additionally, the rise of blockchain technology suggests unprecedented socio-economic benefits for energy systems, especially when coupled with P2P energy trading. Despite such future prospects in energy systems, three key challenges might hinder the full integration of P2P energy trading and blockchain. First, it is quite complicated to design a decentralized P2P market that keeps a fair balance between economic efficiency and information privacy. Secondly, with the proliferation of storage devices, new P2P market designs are needed to account for their inter-temporal dependencies. Thirdly, a practical implementation of blockchain technology for P2P trading is required, which can facilitate efficient trading in a secured and fraud-resilient way, while eliminating any intermediaries’ costs. In this paper, we develop a new decentralized P2P energy trading platform to address all the aforementioned challenges. Our platform consists of two key layers: market and blockchain. The market layer features a parallel and short-term pool-structured auction and is cleared using a novel decentralized Ant-Colony Optimization method. This market arrangement guarantees a near-optimally efficient market solution, preserves players’ privacy, and allows inter-temporal market products trading. The blockchain layer offers a high level of automation, security, and fast real-time settlements through smart contract implementation. Finally, using real-world data, we simulate the functionality of the platform regarding energy trading, market clearing, smart contract operations, and blockchain-based settlements.
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

Our objective in this work is video-text retrieval – in particular a joint embedding that enables efficient text-to-video retrieval. The challenges in this area include the design of the visual architecture and the nature of the training data, in that the available large scale video-text training datasets, such as HowTo100M, are noisy and hence competitive performance is achieved only at scale through large amounts of compute.We address both these challenges in this paper. We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is an adaptation and extension of the recent ViT and Timesformer architectures, and consists of attention in both space and time. The model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. It is trained with a curriculum learning schedule that begins by treating images as ‘frozen’ snapshots of video, and then gradually learns to attend to increasing temporal context when trained on video datasets. We also provide a new video-text pretraining dataset WebVid-2M, comprised of over two million videos with weak captions scraped from the internet. Despite training on datasets that are an order of magnitude smaller, we show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.
A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training

This work develops a continuous sign language (SL) recognition framework with deep neural networks, which directly transcribes videos of SL sentences to sequences of ordered gloss labels. Previous methods dealing with continuous SL recognition usually employ hidden Markov models with limited capacity to capture the temporal information. In contrast, our proposed architecture adopts deep convolutional neural networks with stacked temporal fusion layers as the feature extraction module, and bidirectional recurrent neural networks as the sequence learning module. We propose an iterative optimization process for our architecture to fully exploit the representation capability of deep neural networks with limited data. We first train the end-to-end recognition model for alignment proposal, and then use the alignment proposal as strong supervisory information to directly tune the feature extraction module. This training process can run iteratively to achieve improvements on the recognition performance. We further contribute by exploring the multimodal fusion of RGB images and optical flow in sign language. Our method is evaluated on two challenging SL recognition benchmarks, and outperforms the state of the art by a relative improvement of more than 15% on both databases.
The Second Conversational Intelligence Challenge (ConvAI2)

We describe the setting and results of the ConvAI2 NeurIPS competition that aims to further the state-of-the-art in open-domain chatbots. Some key takeaways from the competition are: (1) pretrained Transformer variants are currently the best performing models on this task, (2) but to improve performance on multi-turn conversations with humans, future systems must go beyond single word metrics like perplexity to measure the performance across sequences of utterances (conversations)—in terms of repetition, consistency and balance of dialogue acts (e.g. how many questions asked vs. answered).
Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images

Nuclei mymargin segmentation is a fundamental task for various computational pathology applications including nuclei morphology analysis, cell type classification, and cancer grading. Deep learning has emerged as a powerful approach to segmenting nuclei but the accuracy of convolutional neural networks (CNNs) depends on the volume and the quality of labeled histopathology data for training. In particular, conventional CNN-based approaches lack structured prediction capabilities, which are required to distinguish overlapping and clumped nuclei. Here, we present an approach to nuclei segmentation that overcomes these challenges by utilizing a conditional generative adversarial network (cGAN) trained with synthetic and real data. We generate a large dataset of H&E training images with perfect nuclei segmentation labels using an unpaired GAN framework. This synthetic data along with real histopathology data from six different organs are used to train a conditional GAN with spectral normalization and gradient penalty for nuclei segmentation. This adversarial regression framework enforces higher-order spacial-consistency when compared to conventional CNN models. We demonstrate that this nuclei segmentation approach generalizes across different organs, sites, patients and disease states, and outperforms conventional approaches, especially in isolating individual and overlapping nuclei.
Pyramid Graph Networks With Connection Attentions for Region-Based One-Shot Semantic Segmentation

One-shot image segmentation aims to undertake the segmentation task of a novel class with only one training image available. The difficulty lies in that image segmentation has structured data representations, which yields a many-to-many message passing problem. Previous methods often simplify it to a one-to-many problem by squeezing support data to a global descriptor. However, a mixed global representation drops the data structure and information of individual elements. In this paper, we propose to model structured segmentation data with graphs and apply attentive graph reasoning to propagate label information from support data to query data. The graph attention mechanism could establish the element-to-element correspondence across structured data by learning attention weights between connected graph nodes. To capture correspondence at different semantic levels, we further propose a pyramid-like structure that models different sizes of image regions as graph nodes and undertakes graph reasoning at different levels. Experiments on PASCAL VOC 2012 dataset demonstrate that our proposed network significantly outperforms the baseline method and leads to new state-of-the-art performance on 1-shot and 5-shot segmentation benchmarks.
GANalyze: Toward Visual Definitions of Cognitive Image Properties

We introduce a framework that uses Generative Adversarial Networks (GANs) to study cognitive properties like memorability. These attributes are of interest because we do not have a concrete visual definition of what they entail. What does it look like for a dog to be more memorable? GANs allow us to generate a manifold of natural-looking images with fine-grained differences in their visual attributes. By navigating this manifold in directions that increase memorability, we can visualize what it looks like for a particular generated image to become more memorable. The resulting ``visual definitions" surface image properties (like ``object size") that may underlie memorability. Through behavioral experiments, we verify that our method indeed discovers image manipulations that causally affect human memory performance. We further demonstrate that the same framework can be used to analyze image aesthetics and emotional valence. ganalyze.csail.mit.edu.
Structure-Preserving Super Resolution With Gradient Guidance

Structures matter in single image super resolution (SISR). Recent studies benefiting from generative adversarial network (GAN) have promoted the development of SISR by recovering photo-realistic images. However, there are always undesired structural distortions in the recovered images. In this paper, we propose a structure-preserving super resolution method to alleviate the above issue while maintaining the merits of GAN-based methods to generate perceptual-pleasant details. Specifically, we exploit gradient maps of images to guide the recovery in two aspects. On the one hand, we restore high-resolution gradient maps by a gradient branch to provide additional structure priors for the SR process. On the other hand, we propose a gradient loss which imposes a second-order restriction on the super-resolved images. Along with the previous image-space loss functions, the gradient-space objectives help generative networks concentrate more on geometric structures. Moreover, our method is model-agnostic, which can be potentially used for off-the-shelf SR networks. Experimental results show that we achieve the best PI and LPIPS performance and meanwhile comparable PSNR and SSIM compared with state-of-the-art perceptual-driven SR methods. Visual results demonstrate our superiority in restoring structures while generating natural SR images.
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification

The experimental landscape in natural language processing for social media is too fragmented.Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction.Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domainspecific data.In this paper, we propose a new evaluation framework (TWEETEVAL) consisting of seven heterogeneous Twitter-specific classification tasks.We also provide a strong set of baselines as starting point, and compare different language modeling pre-training strategies.Our initial experiments show the effectiveness of starting off with existing pretrained generic language models, and continue training them on Twitter corpora.
Improved Breast Cancer Classification Through Combining Graph Convolutional Network and Convolutional Neural Network

In a pilot study to improve detection of malignant lesions in breast mammograms, we aimed to develop a new method called BDR-CNN-GCN, combining two advanced neural networks: (i) graph convolutional network (GCN); and (ii) convolutional neural network (CNN). We utilised a standard 8-layer CNN, then integrated two improvement techniques: (i) batch normalization (BN) and (ii) dropout (DO). Finally, we utilized rank-based stochastic pooling (RSP) to substitute the traditional max pooling. This resulted in BDR-CNN, which is a combination of CNN, BN, DO, and RSP. This BDR-CNN was hybridized with a two-layer GCN, and yielded our BDR-CNN-GCN model which was then utilized for analysis of breast mammograms as a 14-way data augmentation method. As proof of concept, we ran our BDR-CNN-GCN algorithm 10 times on the breast mini-MIAS dataset (containing 322 mammographic images), achieving a sensitivity of 96.20±2.90%, a specificity of 96.00±2.31% and an accuracy of 96.10±1.60%. Our BDR-CNN-GCN showed improved performance compared to five proposed neural network models and 15 state-of-the-art breast cancer detection approaches, proving to be an effective method for data augmentation and improved detection of malignant breast masses.
TDN: Temporal Difference Networks for Efficient Action Recognition

Temporal modeling still remains challenging for action recognition in videos. To mitigate this issue, this paper presents a new video architecture, termed as Temporal Difference Network (TDN), with a focus on capturing multi-scale temporal information for efficient action recognition. The core of our TDN is to devise an efficient temporal module (TDM) by explicitly leveraging a temporal difference operator, and systematically assess its effect on short-term and long-term motion modeling. To fully capture temporal information over the entire video, our TDN is established with a two-level difference modeling paradigm. Specifically, for local motion modeling, temporal difference over consecutive frames is used to supply 2D CNNs with finer motion pattern, while for global motion modeling, temporal difference across segments is incorporated to capture long-range structure for motion feature excitation. TDN provides a simple and principled temporal modeling framework and could be instantiated with the existing CNNs at a small extra computational cost. Our TDN presents a new state of the art on the Something-Something V1 & V2 datasets and is on par with the best performance on the Kinetics-400 dataset. In addition, we conduct in-depth ablation studies and plot the visualization results of our TDN, hopefully providing insightful analysis on temporal difference modeling. We release the code at https://github.com/MCG-NJU/TDN.
MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference. Our approach leverages plane-swept cost volumes (widely used in multi-view stereo) for geometry-aware scene reasoning, and combines this with physically based volume rendering for neural radiance field reconstruction. We train our network on real objects in the DTU dataset, and test it on three different datasets to evaluate its effectiveness and generalizability. Our approach can generalize across scenes (even indoor scenes, completely different from our training scenes of objects) and generate realistic view synthesis results using only three input images, significantly outperforming concurrent works on generalizable radiance field reconstruction. Moreover, if dense images are captured, our estimated radiance field representation can be easily fine-tuned; this leads to fast per-scene reconstruction with higher rendering quality and substantially less optimization time than NeRF.
RIC-Unet: An Improved Neural Network Based on Unet for Nuclei Segmentation in Histology Images

As a prerequisite for cell detection, cell classification, and cancer grading, nuclei segmentation in histology images has attracted wide attention in recent years.It is quite a challenging task due to the diversity in staining procedure, cell morphology, and cell arrangement between different histopathology images, especially with different color contrasts.In this paper, an Unet-based neural network, RIC-Unet (residual-inception-channel attention-Unet), for nuclei segmentation is proposed.The techniques of residual blocks, multi-scale and channel attention mechanism are applied on RIC-Unet to segment nuclei more accurately.RIC-Unet is compared with two traditional segmentation methods: CP and Fiji, two original CNN methods: CNN2, CNN3, and original U-net on The Cancer Genomic Atlas (TCGA) dataset.Besides, in this paper, we use Dice, F1-score, and aggregated Jaccard index to evaluate these methods.The average of RIC-Unet and U-net on these three indicators are 0.8008 versus 0.7844, 0.8278 versus 0.8155, and 0.5635 versus 0.5462.In addition, our method won the third place in the computational precision medicine nuclei segmentation challenge together with MICCAI 2018.
Combating Deepfake Videos Using Blockchain and Smart Contracts

With the rise of artificial intelligence (AI) and deep learning techniques, fake digital contents have proliferated in recent years. Fake footage, images, audios, and videos (known as deepfakes) can be a scary and dangerous phenomenon and can have the potential of altering the truth and eroding trust by giving false reality. Proof of authenticity (PoA) of digital media is critical to help eradicate the epidemic of forged content. Current solutions lack the ability to provide history tracking and provenance of digital media. In this paper, we provide a solution and a general framework using Ethereum smart contracts to trace and track the provenance and history of digital content to its original source even if the digital content is copied multiple times. The smart contract utilizes the hashes of the interplanetary file system (IPFS) used to store digital content and its metadata. Our solution focuses on video content, but the solution framework provided in this paper is generic enough and can be applied to any other form of digital content. Our solution relies on the principle that if the content can be credibly traced to a trusted or reputable source, the content can then be real and authentic. The full code of the smart contract has been made publicly available at Github.
Completely Automated CNN Architecture Design Based on Blocks

The performance of convolutional neural networks (CNNs) highly relies on their architectures. In order to design a CNN with promising performance, extensive expertise in both CNNs and the investigated problem domain is required, which is not necessarily available to every interested user. To address this problem, we propose to automatically evolve CNN architectures by using a genetic algorithm (GA) based on ResNet and DenseNet blocks. The proposed algorithm is completely automatic in designing CNN architectures. In particular, neither preprocessing before it starts nor postprocessing in terms of CNNs is needed. Furthermore, the proposed algorithm does not require users with domain knowledge on CNNs, the investigated problem, or even GAs. The proposed algorithm is evaluated on the CIFAR10 and CIFAR100 benchmark data sets against 18 state-of-the-art peer competitors. Experimental results show that the proposed algorithm outperforms the state-of-the-art CNNs hand-crafted and the CNNs designed by automatic peer competitors in terms of the classification performance and achieves a competitive classification accuracy against semiautomatic peer competitors. In addition, the proposed algorithm consumes much less computational resource than most peer competitors in finding the best CNN architectures.
A generic communication scheduler for distributed DNN training acceleration

We present ByteScheduler, a generic communication scheduler for distributed DNN training acceleration. ByteScheduler is based on our principled analysis that partitioning and rearranging the tensor transmissions can result in optimal results in theory and good performance in real-world even with scheduling overhead. To make ByteScheduler work generally for various DNN training frameworks, we introduce a unified abstraction and a Dependency Proxy mechanism to enable communication scheduling without breaking the original dependencies in framework engines. We further introduce a Bayesian Optimization approach to auto-tune tensor partition size and other parameters for different training models under various networking conditions. ByteScheduler now supports TensorFlow, PyTorch, and MXNet without modifying their source code, and works well with both Parameter Server (PS) and all-reduce architectures for gradient synchronization, using either TCP or RDMA. Our experiments show that ByteScheduler accelerates training with all experimented system configurations and DNN models, by up to 196% (or 2.96X of original speed).