Reviewerly/exHarmony huggingface dataset:
Static and Dynamic Concepts for Self-supervised Video Representation Learning

In this paper, we propose a novel learning scheme for self-supervised video representation learning. Motivated by how humans understand videos, we propose to first learn general visual concepts then attend to discriminative local areas for video understanding. Specifically, we utilize static frame and frame difference to help decouple static and dynamic concepts, and respectively align the concept distributions in latent space. We add diversity and fidelity regularizations to guarantee that we learn a compact set of meaningful concepts. Then we employ a cross-attention mechanism to aggregate detailed local features of different concepts, and filter out redundant concepts with low activations to perform local concept contrast. Extensive experiments demonstrate that our method distills meaningful static and dynamic concepts to guide video understanding, and obtains state-of-the-art results on UCF-101, HMDB-51, and Diving-48.
Interpretable Privacy-Preserving Collaborative Deep Learning for Taxi Trip Duration Forecasting

Traffic data are obtained from various distributed sources such as infrastructure and vehicle sensors developed by various organisations, and often cannot be processed together because of data privacy regulations. Thus, distributed machine learning methods are required to process the data without sharing them. Federated learning allows the processing of data distributed by transmitting only the parameters without sharing the real data. The federated learning architecture is based mainly on deep learning, which is often more accurate than other machine learning approaches. However, deep-learning-based models are black-box models, and should be explained to increase trust in the system for both users and developers. Despite the fact that various explainability methods have been proposed, the solutions for explainable federated models are insufficient. In this study, we used a federated deep learning model to predict a taxi trip duration within Brunswick region. We showed situations for which federated learning improves the prediction quality, allowing for an accuracy comparable to that obtained on the complete dataset. Moreover, we investigated how the amount of transmitted information in federated learning can be optimised while maintaining the same accuracy. Finally, we propose how the federated deep learning model can be interpreted using explainability methods without transmitting raw data and compare the results of various explainability approaches.
Work Together: Correlation-Identity Reconstruction Hashing for Unsupervised Cross-Modal Retrieval

Unsupervised cross-modal hashing has attracted considerable attention to support large-scale cross-modal retrieval. Although promising progresses have been made so far, existing methods still suffer from limited capability on excavating and preserving the intrinsic multi-modal semantics. In this paper, we propose a Correlation-Identity Reconstruction Hashing (CIRH) method to alleviate this challenging problem. We develop a new unsupervised deep cross-modal hash learning framework to model and preserve the heterogeneous multi-modal correlation semantics into both hash codes and functions, and simultaneously, we involve both the hash codes and functions with the descriptive identity semantics. Specifically, we construct a multi-modal collaborated graph to model the heterogeneous multi-modal correlations, and jointly perform the intra-modal and cross-modal semantic aggregation on homogeneous and heterogeneous graph networks to generate a multi-modal complementary representation with correlation reconstruction. Furthermore, an identity semantic reconstruction process is designed to involve the generated representation with identity semantics by reconstructing the input modality representations. Finally, we propose a correlation-identity consistent hash function learning strategy to transfer the modelled multi-modal semantics into the neural networks of modality-specific deep hash functions. Experiments demonstrate the superior performance of the proposed method on both retrieval accuracy and efficiency. We provide our source codes and experimental datasets at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/XizeWu/CIRH</uri>
PANDORA: A Panoramic Detection Dataset for Object with Orientation

Panoramic images have become increasingly popular as omnidirectional panoramic technology has advanced. Many datasets and works resort to object detection to better understand the content of the panoramic image. These datasets and detectors use a Bounding Field of View (BFoV) as a bounding box in panoramic images. However, we observe that the object instances in panoramic images often appear with arbitrary orientations. It indicates that BFoV as a bounding box is inappropriate, limiting the performance of detectors. This paper proposes a new bounding box representation, Rotated Bounding Field of View (RBFoV), for the panoramic image object detection task. Then, based on the RBFoV, we present a PANoramic Detection dataset for Object with oRientAtion (PANDORA). Finally, based on PANDORA, we evaluate the current state-of-the-art panoramic image object detection methods and design an anchor-free object detector called R-CenterNet for panoramic images. Compared with these baselines, our R-CenterNet shows its advantages in terms of detection performance. Our PANDORA dataset and source code are available at https://github.com/tdsuper/SphericalObjectDetection .
P<sup>3</sup>IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision

In this paper, we study the problem of procedure planning in instructional videos. Here, an agent must produce a plausible sequence of actions that can transform the environment from a given start to a desired goal state. When learning procedure planning from instructional videos, most recent work leverages intermediate visual observations as supervision, which requires expensive annotation efforts to localize precisely all the instructional steps in training videos. In contrast, we remove the need for expensive temporal video annotations and propose a weakly supervised approach by learning from natural language instructions. Our model is based on a transformer equipped with a memory module, which maps the start and goal observations to a sequence of plausible actions. Furthermore, we augment our model with a probabilistic generative module to capture the uncertainty inherent to procedure planning, an aspect largely overlooked by previous work. We evaluate our model on three datasets and show our weakly-supervised approach outperforms previous fully supervised state-of-the-art models on multiple metrics.
Instance Switching-Based Contrastive Learning for Fine-Grained Airplane Detection

Detecting airplanes from high-resolution remote sensing images has a variety of applications. The characteristics of clear details, rich spatial and texture information of objects in high-resolution remote sensing images make it possible to identify different types of airplanes from backgrounds. However, airplanes usually exhibit slight inter-class discrepancy and unbalanced class distribution, which pose significant challenges to fine-grained detection of airplanes. In this paper, we propose the ISCL, an Instance Switching-based Contrastive Learning method for fine-grained airplane detection. Specifically, we introduce a Contrastive Learning-based Module (CLM) to widen the inter-class distance while narrowing the intra-class distance by optimizing feature space distribution with the InfoNCE <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">+</sup> loss, which is built on a serial head in a cascaded way. Then, we design a Refined Instance Switching (ReIS) module to alleviate the class imbalance problem. To take full advantage of the CLM and ReIS, we further introduce an optimization strategy which is an organic combination of the two modules to widen the distances of different airplane categories that are easily confused. In addition, we contribute a fine-grained attribute-assisted dataset, dubbed GF-RarePlanes Dataset (GRD), to help the detectors better learn the subtle differences between the airplanes. Extensive experiments on two datasets ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., GF and FAIR1M) demonstrate that our proposed method can significantly improve the accuracy of fine-grained airplane detection under both HBB and OBB scenarios. Dataset and codes will be available at https://lanxin1011.github.io/ISCL/.
A Cloud-Based Predictive Model for the Detection of Breast Cancer

Invasive cancer is the biggest cause of death worldwide, especially among women. Early cancer detection is vital to health. Early identification of breast cancer improves prognosis and survival odds by allowing for timely clinical therapy. For accurate cancer prediction, machine learning requires quick analytics and feature extraction. Cloud-based machine learning is vital for illness diagnosis in rural areas with few medical facilities. In this research, random forests, logistic regression, decision trees, and SVM are employed, and the authors assess the performance of various algorithms using confusion measures and AUROC to choose the best machine learning model for breast cancer prediction. Precision, recall, accuracy, and specificity are used to calculate results. Confusion matrix is based on predicted cases. The ML model's performance is evaluated. For simulation, the authors used the Wisconsin Dataset of Breast Cancer (WDBC). Through experiments, it can be seen that the SVM model reached 98.24% accuracy with an AUC of 0.993, while the logistic regression achieved 94.54% accuracy with an AUC of 0.998.
Improving Adversarial Robustness of 3D Point Cloud Classification Models

3D point cloud classification models based on deep neural networks were proven to be vulnerable to adversarial examples, with a quantity of novel attack techniques proposed by researchers recently. It is of paramount importance to preserve the robustness of 3D models under adversarial environments, considering their broad application in safety- and security-critical tasks. Unfortunately, existing defenses are not general enough to satisfactorily mitigate all types of attacks. In this paper, we design two innovative methodologies to improve the adversarial robustness of 3D point cloud classification models. (1) We introduce CCN, a novel point cloud architecture which can smooth and disrupt the adversarial perturbations. (2) We propose AMS, a novel data augmentation strategy to adaptively balance the model usability and robustness. Extensive evaluations indicate the integration of the two techniques provides much more robustness than existing defense solutions for 3D classification models. Our code can be found in https://github.com/GuanlinLee/CCNAMS .
SynShine: Improved Fixing of Syntax Errors

Novice programmers struggle with the complex syntax of modern programming languages like <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Java</small> , and make lot of syntax errors. The diagnostic syntax error messages from compilers and IDEs are sometimes useful, but often the messages are cryptic and puzzling. Novices could be helped, and instructors’ time saved, by automated repair suggestions when dealing with syntax errors. Large samples of novice errors and fixes are now available, offering the possibility of data-driven machine-learning approaches to help novices fix syntax errors. Current machine-learning approaches do a reasonable job fixing syntax errors in shorter programs, but don't work as well even for moderately longer programs. We introduce <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SynShine</small> , a machine-learning based tool that substantially improves on the state-of-the-art, by learning to use compiler diagnostics, employing a very large neural model that leverages unsupervised pre-training, and relying on multi-label classification rather than autoregressive synthesis to generate the (repaired) output. We describe <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SynShine</small> 's architecture in detail, and provide a detailed evaluation. We have built <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SynShine</small> into a free, open-source version of Visual Studio Code (VSCode); we make all our source code and models freely available.
Filter Clustering for Compressing CNN Model With Better Feature Diversity

As a practical approach for compressing convolutional neural networks (CNNs), network pruning has been rapidly developed in recent years. The conventional methods prune inactive filters permanently from models to reduce the width of each layer and then train the pruned model until convergence. However, such methods have limitations in that: (1) The activation-based pruning criteria ignore the correlation between filters, leading to attenuation in types of features; (2) The permanent filter removal restricts the architecture of models in the subsequent training so that reducing the chances of learning more features; (3) The single-width compression may generate narrow layers that block the information flow, resulting in limited feature capacity in the next layers and hard optimization. These limitations reduce the feature diversity in the pruned model and thus lead to sub-optimal model quality. In this paper, a compression method named filter clustering is proposed to rectify the problem of poor feature diversity in traditional pruning and achieve better model quality from three perspectives. Firstly, to maintain the variety of features after pruning, we treat the model compression as a clustering task and merge filters with similar outputs, rather than removing inactive filters. Specifically, a handy estimation approach is designed to convert the similarity of the output into filter similarity, which liberates the measurement from sampling numerous images. Secondly, to increase the probability of learning more features during training, we propose a periodic training and clustering pipeline, which creates a larger optimization space by dynamically exploring different sub-model architectures. Finally, to prevent the feature capacity from being influenced by the narrow layers, we introduce and leverage a fusible anti-blocking branch to smoothly remove such layers. Extensive experiments demonstrate that the proposed method can achieve compact models with better feature diversity and reduce 1%~15% more calculations than the previous methods while maintaining performance.
Toward Systematic Design Considerations of Organizing Multiple Views

Multiple-view visualization (MV) has been used for visual analytics in various fields (e.g., bioinformatics, cybersecurity, and intelligence analysis). Because each view encodes data from a particular per-spective, analysts often use a set of views laid out in 2D space to link and synthesize information. The difficulty of this process is impacted by the spatial organization of these views. For instance, connecting information from views far from each other can be more challenging than neighboring ones. However, most visual analysis tools currently either fix the positions of the views or completely delegate this organization of views to users (who must manually drag and move views). This either limits user involvement in managing the layout of MV or is overly flexible without much guidance. Then, a key design challenge in MV layout is determining the factors in a spatial organization that impact understanding. To address this, we review a set of MV-based systems and identify considerations for MV layout rooted in two key concerns: perception, which considers how users perceive view relationships, and content, which considers the relationships in the data. We show how these allow us to study and analyze the design of MV layout systematically.
Extremely Low-Light Image Enhancement with Scene Text Restoration

Deep learning-based methods have made impressive progress in enhancing extremely low-light images - the image quality of the reconstructed images has generally improved. However, we found out that most of these methods could not sufficiently recover the image details, for instance, the texts in the scene. In this paper, a novel image enhancement framework is proposed to precisely restore the scene texts, as well as the overall quality of the image simultaneously under extremely low-light conditions. Mainly, we employed a self-regularised attention map, an edge map, and a novel text detection loss. In addition, leveraging the synthetic low-light images is beneficial for image enhancement on the genuine ones in terms of text detection. The quantitative and qualitative experimental results have shown that the proposed model outperforms state-of-the-art methods in image restoration, text detection, and text spotting on See In the Dark and ICDAR15 datasets.
GVD-net: Graph embedding-based Machine Learning Model for Smart Contract Vulnerability Detection

Ethereum smart contract and the agreements contained therein exist across a distributed, decentralized blockchain network, which controls the execution of Dapps, and transactions are trackable and irreversible. Since the amount of smart contracts on Ethereum has a rapid growth, which also lead to a large number of security vulnerabilities. However, the traditional method such as static symbolic analysis requires manual analysis in advance with high false detection rate. In this paper, we propose a GVD-net model for smart contract vulnerability detection. GVD-net consists of preprocessing section,backbone-net and detection section.We prepossess the source code and gain a weight matrix in the first section. In backbone-net, we generate a CFG graph according to the variables and function calling relationships of the solidity code, and use the CFG graph to build the corresponding relationship of nodes and generate a non-Euclidean graph. Then we use the adjacent search algorithm to generate a non-complete random walk sequence, and use the graph embedding algorithm Node2Vec to generate a 256-dimensional vector and gain another weight matrix. We input the test code into GVD-net to generate the threshold ϑ in the final section and will classify the smart contract as danger code if the ϑ is bigger than 0.75. Moreover, GVD-net will confirm the type of vulnerability according to weight matrix. We conduct comprehensive experiments on GVD-net with the SBcurated data-set, and the experiment results show that (1) GVD-net can classify a smart contract as safe or danger code, with the accuracy of 90.2% (2) GVD-net can detect three types of vulnerability (arithmetic issues, access control and asset frozen) and attain good performance within 1s.
Unimodal-Concentrated Loss: Fully Adaptive Label Distribution Learning for Ordinal Regression

Learning from a label distribution has achieved promising results on ordinal regression tasks such as facial age and head pose estimation wherein, the concept of adaptive label distribution learning (ALDL) has drawn lots of attention recently for its superiority in theory. However, compared with the methods assuming fixed form label distribution, ALDL methods have not achieved better performance. We argue that existing ALDL algorithms do not fully exploit the intrinsic properties of ordinal regression. In this paper, we emphatically summarize that learning an adaptive label distribution on ordinal regression tasks should follow three principles. First, the probability corresponding to the ground-truth should be the highest in label distribution. Second, the probabilities of neighboring labels should decrease with the increase of distance away from the ground-truth, i.e., the distribution is unimodal. Third, the label distribution should vary with samples changing, and even be distinct for different instances with the same label, due to the different levels of difficulty and ambiguity. Under the premise of these principles, we propose a novel loss function for fully adaptive label distribution learning, namely unimodal-concentrated loss. Specifically, the unimodal loss derived from the learning to rank strategy constrains the distribution to be unimodal. Furthermore, the estimation error and the variance of the predicted distribution for a specific sample are integrated into the proposed concentrated loss to make the predicted distribution maximize at the ground-truth and vary according to the predicting uncertainty. Extensive experimental results on typical ordinal regression tasks including age and head pose estimation, show the superiority of our proposed unimodal-concentrated loss compared with existing loss functions.
Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection

Cancers can have highly heterogeneous uptake patterns best visualised in positron emission tomography. These patterns are essential to detect, diagnose, stage and predict the evolution of cancer. Due to this heterogeneity, a general-purpose cancer detection model can be built using unsupervised learning anomaly detection models; these models learn a healthy representation of tissue and detect cancer by predicting deviations from healthy appearances. This task alone requires models capable of accurately learning long-range interactions between organs, imaging patterns, and other abstract features with high levels of expressivity. Such characteristics are suitably satisfied by transformers, and have been shown to generate state-of-the-art results in unsupervised anomaly detection by training on healthy data. This work expands upon such approaches by introducing multi-modal conditioning of the transformer via cross-attention, i.e. supplying anatomical reference information from paired CT images to aid the PET anomaly detection task. Using 83 whole-body PET/CT samples containing various cancer types, we show that our anomaly detection method is robust and capable of achieving accurate cancer localisation results even in cases where healthy training data is unavailable. Furthermore, the proposed model uncertainty, in conjunction with a kernel density estimation approach, is shown to provide a statistically robust alternative to residual-based anomaly maps. Overall, a superior performance is demonstrated against leading alternatives, drawing attention to the potential of these approaches.
A Multi-Object Tracking Algorithm With Center-Based Feature Extraction and Occlusion Handling

For tracking suspicious objects using intelligent robots, Multiple Object Tracking (MOT) has gained great attention. MOT is easily affected by long-term severe occlusion. This work proposes a joint MOT algorithm to handle such occlusion. Pairs of frames in complicated environments are taken as input. A center-based feature extraction framework is designed for precisely detecting objects and extracting their feature maps. A ConvGRU module is applied to learn permanent representations by using historical spatio-temporal information of objects. A Hungarian matching method is applied to match the detected objects and predicted predictions. The proposed algorithm is compared with several representative methods on two public multi-object tracking benchmarks. Furthermore, this work constructs a database with videos captured from street scenarios and uses it to test the proposed algorithm and its peers. Experimental results demonstrate that the proposed algorithm outperforms its peers, especially under long-term severe occlusion, thus advancing the field of MOT.