Reviewerly/exHarmony huggingface dataset:
Learning Dual-Pooling Graph Neural Networks for Few-Shot Video Classification

We address the problem of few-shot video classification that learns classifiers for novel concepts from only a few examples. Most current methods ignore to explicitly consider the relations in both intra-video and inter-video domains, thus cannot take full advantage of the structural information in few-shot learning. In this paper, we propose to exploit the comprehensive intra-video and inter-video relations via Graph Neural Networks (GNNs). To improve the discriminative ability for accurately selecting the representative video content and refining video relations, a Dual-Pooling GNN (DPGNN) is constructed, which stacks customized graph pooling layers in a hierarchical fashion. Specifically, to select the most representative frames in a video, we build intra-video graphs and utilize a node pooling module to extract robust video-level features. We construct an inter-video graph by taking the video-level features as nodes. By designing an edge pooling module, the proposed method can adaptively eliminate the negative relations in the inter-video graph. Extensive experimental results show that our method consistently outperforms the state-of-the-art on two benchmarks.
Human Activity Recognition Across Scenes and Categories Based on CSI

Activity recognition based on channel state information (CSI) plays an increasingly important role in human computer interaction. However most CSI activity recognition systems need to re-collect a large amount of samples and retrain model when they are used in new environments or recognize new types of activities, which greatly reduces the practicality of CSI activity recognition. To address this problem we design an adaptable CSI activity recognition system based on meta-learning, which only needs to fine-tune model with very little train effort when it is used in new environments or recognize new types of activities. Specifically, we first use meta-learning algorithm to get the pre-trained model that adapts to task distribution, when the environment or activity category changes, our system doesn't need to retrain the model and has maximal performance after updates the pre-trained model through one or more gradient steps computed with a small amount of samples from new activities. To prevent the loss of CSI time information after feature extraction with multi-layer CNN, we add time encoding on CSI data as the input of CNN neural network. Considering that CSI data may be labeled incorrectly during labeling process, we improve categorical cross entropy loss(CCE) to enhance the system's robustness to these mislabeled data. We test our system on the gesture dataset and the body activity dataset, and the experimental results show that our system achieves average accuracy of 72 percent with one sample of each new activity and 89.6 percent with five samples of each new activity.
Decentralized Multi-authority ABE for DNFs from LWE

We construct the first decentralized multi-authority attribute-based encryption ( $$\mathsf {MA}\text {-}\mathsf {ABE}$$ ) scheme for a non-trivial class of access policies whose security is based (in the random oracle model) solely on the Learning With Errors (LWE) assumption. The supported access policies are ones described by $$\mathsf {DNF}$$ formulas. All previous constructions of $$\mathsf {MA}\text {-}\mathsf {ABE}$$ schemes supporting any non-trivial class of access policies were proven secure (in the random oracle model) assuming various assumptions on bilinear maps. In our system, any party can become an authority and there is no requirement for any global coordination other than the creation of an initial set of common reference parameters. A party can simply act as a standard ABE authority by creating a public key and issuing private keys to different users that reflect their attributes. A user can encrypt data in terms of any $$\mathsf {DNF}$$ formulas over attributes issued from any chosen set of authorities. Finally, our system does not require any central authority. In terms of efficiency, when instantiating the scheme with a global bound s on the size of access policies, the sizes of public keys, secret keys, and ciphertexts, all grow with s. Technically, we develop new tools for building ciphertext-policy ABE ( $$\mathsf {CP}\text {-}\mathsf {ABE}$$ ) schemes using LWE. Along the way, we construct the first provably secure $$\mathsf {CP}\text {-}\mathsf {ABE}$$ scheme supporting access policies in $$\mathsf {NC}^1$$ under the LWE assumption that avoids the generic universal-circuit-based key-policy to ciphertext-policy transformation. In particular, our construction relies on linear secret sharing schemes with new properties and in some sense is more similar to $$\mathsf {CP}\text {-}\mathsf {ABE}$$ schemes that rely on bilinear maps. While our $$\mathsf {CP}\text {-}\mathsf {ABE}$$ construction is not more efficient than existing ones, it is conceptually intriguing and further we show how to extend it to get the $$\mathsf {MA}\text {-}\mathsf {ABE}$$ scheme described above.
Machine learning techniques for mitoses classification

Pathologists analyze biopsy material at both the cellular and structural level to determine diagnosis and cancer stage. Mitotic figures are surrogate biomarkers of cellular proliferation that can provide prognostic information; thus, their precise detection is an important factor for clinical care. Convolutional Neural Networks (CNNs) have shown remarkable performance on several recognition tasks. Utilizing CNNs for mitosis classification may aid pathologists to improve the detection accuracy.We studied two state-of-the-art CNN-based models, ESPNet and DenseNet, for mitosis classification on six whole slide images of skin biopsies and compared their quantitative performance in terms of sensitivity, specificity, and F-score. We used raw RGB images of mitosis and non-mitosis samples with their corresponding labels as training input. In order to compare with other work, we studied the performance of these classifiers and two other architectures, ResNet and ShuffleNet, on the publicly available MITOS breast biopsy dataset and compared the performance of all four in terms of precision, recall, and F-score (which are standard for this data set), architecture, training time and inference time.The ESPNet and DenseNet results on our primary melanoma dataset had a sensitivity of 0.976 and 0.968, and a specificity of 0.987 and 0.995, respectively, with F-scores of .968 and .976, respectively. On the MITOS dataset, ESPNet and DenseNet showed a sensitivity of 0.866 and 0.916, and a specificity of 0.973 and 0.980, respectively. The MITOS results using DenseNet had a precision of 0.939, recall of 0.916, and F-score of 0.927. The best published result on MITOS (Saha et al. 2018) reported precision of 0.92, recall of 0.88, and F-score of 0.90. In our architecture comparisons on MITOS, we found that DenseNet beats the others in terms of F-Score (DenseNet 0.927, ESPNet 0.890, ResNet 0.865, ShuffleNet 0.847) and especially Recall (DenseNet 0.916, ESPNet 0.866, ResNet 0.807, ShuffleNet 0.753), while ResNet and ESPNet have much faster inference times (ResNet 6 s, ESPNet 8 s, DenseNet 31 s). ResNet is faster than ESPNet, but ESPNet has a higher F-Score and Recall than ResNet, making it a good compromise solution.We studied several state-of-the-art CNNs for detecting mitotic figures in whole slide biopsy images. We evaluated two CNNs on a melanoma cancer dataset and then compared four CNNs on a public breast cancer data set, using the same methodology on both. Our methodology and architecture for mitosis finding in both melanoma and breast cancer whole slide images has been thoroughly tested and is likely to be useful for finding mitoses in any whole slide biopsy images.
Simultaneous Detection and Tracking with Motion Modelling for Multiple Object Tracking

Deep learning based Multiple Object Tracking (MOT) currently relies on off-the-shelf detectors for tracking-by-detection. This results in deep models that are detector biased and evaluations that are detector influenced. To resolve this issue, we introduce Deep Motion Modeling Network (DMM-Net) that can estimate multiple objects’ motion parameters to perform joint detection and association in an end-to-end manner. DMM-Net models object features over multiple frames and simultaneously infers object classes, visibility and their motion parameters. These outputs are readily used to update the tracklets for efficient MOT. DMM-Net achieves PR-MOTA score of 12.80 @ 120+ fps for the popular UA-DETRAC challenge - which is better performance and orders of magnitude faster. We also contribute a synthetic large-scale public dataset Omni-MOT for vehicle tracking that provides precise ground-truth annotations to eliminate the detector influence in MOT evaluation. This 14M+ frames dataset is extendable with our public script (Code at Dataset , Dataset Recorder , Omni-MOT Source ). We demonstrate the suitability of Omni-MOT for deep learning with DMM-Net, and also make the source code of our network public.
Mononizing binocular videos

This paper presents the idea of mono-nizing binocular videos and a framework to effectively realize it. Mono-nize means we purposely convert a binocular video into a regular monocular video with the stereo information implicitly encoded in a visual but nearly-imperceptible form. Hence, we can impartially distribute and show the mononized video as an ordinary monocular video. Unlike ordinary monocular videos, we can restore from it the original binocular video and show it on a stereoscopic display. To start, we formulate an encoding-and-decoding framework with the pyramidal deformable fusion module to exploit long-range correspondences between the left and right views, a quantization layer to suppress the restoring artifacts, and the compression noise simulation module to resist the compression noise introduced by modern video codecs. Our framework is self-supervised, as we articulate our objective function with loss terms defined on the input: a monocular term for creating the mononized video, an invertibility term for restoring the original video, and a temporal term for frame-to-frame coherence. Further, we conducted extensive experiments to evaluate our generated mononized videos and restored binocular videos for diverse types of images and 3D movies. Quantitative results on both standard metrics and user perception studies show the effectiveness of our method.
SumGraph: Video Summarization via Recursive Graph Modeling

The goal of video summarization is to select keyframes that are visually diverse and can represent a whole story of an input video. State-of-the-art approaches for video summarization have mostly regarded the task as a frame-wise keyframe selection problem by aggregating all frames with equal weight. However, to find informative parts of the video, it is necessary to consider how all the frames of the video are related to each other. To this end, we cast video summarization as a graph modeling problem. We propose recursive graph modeling networks for video summarization, termed SumGraph, to represent a relation graph, where frames are regarded as nodes and nodes are connected by semantic relationships among frames. Our networks accomplish this through a recursive approach to refine an initially estimated graph to correctly classify each node as a keyframe by reasoning the graph representation via graph convolutional networks. To leverage SumGraph in a more practical environment, we also present a way to adapt our graph modeling in an unsupervised fashion. With SumGraph, we achieved state-of-the-art performance on several benchmarks for video summarization in both supervised and unsupervised manners.
DVI: Depth Guided Video Inpainting for Autonomous Driving

To get clear street-view and photo-realistic simulation in autonomous driving, we present an automatic video inpainting algorithm that can remove traffic agents from videos and synthesize missing regions with the guidance of depth/point cloud. By building a dense 3D map from stitched point clouds, frames within a video are geometrically correlated via this common 3D map. In order to fill a target inpainting area in a frame, it is straightforward to transform pixels from other frames into the current one with correct occlusion. Furthermore, we are able to fuse multiple videos through 3D point cloud registration, making it possible to inpaint a target video with multiple source videos. The motivation is to solve the long-time occlusion problem where an occluded area has never been visible in the entire video. To our knowledge, we are the first to fuse multiple videos for video inpainting. To verify the effectiveness of our approach, we build a large inpainting dataset in the real urban road environment with synchronized images and Lidar data including many challenge scenes, e.g., long time occlusion. The experimental results show that the proposed approach outperforms the state-of-the-art approaches for all the criteria, especially the RMSE (Root Mean Squared Error) has been reduced by about $$\mathbf{13} \%$$ .
Appearance Consensus Driven Self-supervised Human Mesh Recovery

We present a self-supervised human mesh recovery framework to infer human pose and shape from monocular images in the absence of any paired supervision. Recent advances have shifted the interest towards directly regressing parameters of a parametric human model by supervising them on large-scale datasets with 2D landmark annotations. This limits the generalizability of such approaches to operate on images from unlabeled wild environments. Acknowledging this we propose a novel appearance consensus driven self-supervised objective. To effectively disentangle the foreground (FG) human we rely on image pairs depicting the same person (consistent FG) in varied pose and background (BG) which are obtained from unlabeled wild videos. The proposed FG appearance consistency objective makes use of a novel, differentiable Color-recovery module to obtain vertex colors without the need for any appearance network; via efficient realization of color-picking and reflectional symmetry. We achieve state-of-the-art results on the standard model-based 3D pose estimation benchmarks at comparable supervision levels. Furthermore, the resulting colored mesh prediction opens up the usage of our framework for a variety of appearance-related tasks beyond the pose and shape estimation, thus establishing our superior generalizability.
Topology Learning-Based Fuzzy Random Neural Networks for Streaming Data Regression

As a type of evolving-fuzzy system, the evolving-fuzzy-neuro (EFN) system uses the structure inspired by neural networks to determine its parameters (fuzzy sets and fuzzy rules), so EFN system can inherit the advantages of neural networks. However, for streaming data regression, EFN systems still have several drawbacks: determining fuzzy sets is not robust to data sequence; determining fuzzy rules is complex as subspaces that can approximate to a Takagi–Sugeno–Kang (TSK) rule need to be obtained, and many parameters need to be optimized; and it is difficult to detect and adapt to changes in the data distribution, i.e., concept drift, if the output is a continuous variable. Hence, in this article, a novel evolving-fuzzy-neuro system, called the topology learning-based fuzzy random neural network (TLFRNN), is proposed. In TLFRNN, an online topology learning algorithm is designed to self-organize each layer of TLFRNN. Different from current EFN systems, TLFRNN learns multiple fuzzy sets to reduce the impact of noises on each fuzzy set, and a randomness layer is designed, which assigning the probability of each fuzzy set. Also, TLFRNN does not utilize TSK rules; instead uses a simple inference that considers fuzzy and random information of data simultaneously. More importantly, in TLFRNN, concept drift can be detected and adapted easily and rapidly. The experiments demonstrate that TLFRNN achieves superior performance compared to other EFSs.
In-Home Daily-Life Captioning Using Radio Signals

This paper aims to caption daily life – i.e., to create a textual description of people’s activities and interactions with objects in their homes. Addressing this problem requires novel methods beyond traditional video captioning, as most people would have privacy concerns about deploying cameras throughout their homes. We introduce RF-Diary, a new model for captioning daily life by analyzing the privacy-preserving radio signal in the home with the home’s floormap. RF-Diary can further observe and caption people’s life through walls and occlusions and in dark settings. In designing RF-Diary, we exploit the ability of radio signals to capture people’s 3D dynamics, and use the floormap to help the model learn people’s interactions with objects. We also use a multi-modal feature alignment training scheme that leverages existing video-based captioning datasets to improve the performance of our radio-based captioning model. Extensive experimental results demonstrate that RF-Diary generates accurate captions under visible conditions. It also sustains its good performance in dark or occluded settings, where video-based captioning approaches fail to generate meaningful captions.(For more information, please visit our project webpage: http://rf-diary.csail.mit.edu )
Help first-year college students to learn their library through an augmented reality game

This paper presents a reflection on a pilot Augmented Reality (AR) Treasure Hunt created using a free AR application that served as a library orientation activity for first year college students. This game sought to aid student wayfinding in the library by directing them to locate puzzles in the library building that helped them reach the goal of solving a digital treasure box. Students reported that this game was informative and creative, and was received positively. Students commented that the AR game helped familiarize them with library spaces, services, and subject librarians. Inquiry into the students' experiences with the activity concluded that the gameplay of the AR gave allowed them to become more comfortable with library spaces. Due to the accessibility of the platform, replicating an AR game similar to the pilot game is possible for anyone interested in providing an immersive AR experience for a library orientation or an entry level library instruction class.
Human activity recognition by manifold regularization based dynamic graph convolutional networks

Deep learning has shown superiority to extract more representative features from multimedia data in recent years. Recently, the most typical graph convolutional networks (GCN) has achieved excellent performance in the semi-supervised framework-based data representation learning tasks. GCN successfully generalizes traditional convolutional neural networks to encode arbitrary graphs by exploiting the graph Laplacian-based sample structure information. However, GCN only fuses the static structure information. It is difficult to guarantee that its structure information is optimal during the training process and applicable for all practical applications. To tackle the above problem, in this paper, we propose a manifold regularized dynamic graph convolutional network (MRDGCN). The proposed MRDGCN automatically updates the structure information by manifold regularization until model fitting. In particular, we build an optimization convolution layer formulation to acquire the optimal structure information. Thus, MRDGCN can automatically learn high-level sample features to improve the performance of data representation learning. To demonstrate the effectiveness of our proposed model, we apply MRDGCN on the semi-supervised classification tasks. The extensive experiment results on human activity datasets and citation network datasets validate the performance of MRDGCN compared with GCN and other semi-supervised learning methods.
One-Pixel Signature: Characterizing CNN Models for Backdoor Detection

We tackle the convolution neural networks (CNNs) backdoor detection problem by proposing a new representation called one-pixel signature. Our task is to detect/classify if a CNN model has been maliciously inserted with an unknown Trojan trigger or not. We design the one-pixel signature representation to reveal the characteristics of both clean and backdoored CNN models. Here, each CNN model is associated with a signature that is created by generating, pixel-by-pixel, an adversarial value that is the result of the largest change to the class prediction. The one-pixel signature is agnostic to the design choice of CNN architectures, and how they were trained. It can be computed efficiently for a black-box CNN model without accessing the network parameters. Our proposed one-pixel signature demonstrates a substantial improvement (by around 30% in the absolute detection accuracy) over the existing competing methods for backdoored CNN detection/classification. One-pixel signature is a general representation that can be used to characterize CNN models beyond backdoor detection.
Task Assignment of UAV Swarm Based on Wolf Pack Algorithm

To perform air missions with an unmanned aerial vehicle (UAV) swarm is a significant trend in warfare. The task assignment among the UAV swarm is one of the key issues in such missions. This paper proposes PSO-GA-DWPA (discrete wolf pack algorithm with the principles of particle swarm optimization and genetic algorithm) to solve the task assignment of a UAV swarm with fast convergence speed. The PSO-GA-DWPA is confirmed with three different ground-attack scenarios by experiments. The comparative results show that the improved algorithm not only converges faster than the original WPA and PSO, but it also exhibits excellent search quality in high-dimensional space.
Differentially private user-based collaborative filtering recommendation based on<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" id="d1e297" altimg="si4.svg"><mml:mi>k</mml:mi></mml:math>-means clustering

Collaborative filtering (CF) recommendation is well-known for its outstanding recommendation performance, but previous researches showed that it could cause privacy leakage for users due to k-nearest neighboring (KNN) attacks. Recently, the notion of differential privacy (DP) has been applied to privacy preservation in recommendation systems. However, as far as we know, existing differentially private CF recommendation systems degrade the recommendation performance (such as recall and precision) to an unacceptable level. In this paper, to address the performance degradation problem, we propose a differentially private user-based CF recommendation system based on k-means clustering (KDPCF). Specifically, to improve the recommendation performance, KDPCF first clusters the dataset into categories by k-means clustering and appropriately adjusts the size of the target category to which the target user belongs, so that only users in the well-sized target category are used for recommendation. Then, it selects efficiently a set of neighbors from the target category at one time by employing only one instance of exponential mechanism instead of the composition of multiple ones, and then uses a CF algorithm to recommend based on this set of neighbors. We theoretically prove that our system achieves differential privacy. Empirically, we use two public datasets to evaluate our recommendation system. The experimental results demonstrate that our system has a significant performance improvement compared to existing ones.