Reviewerly/exHarmony huggingface dataset:
Tangent Images for Mitigating Spherical Distortion

In this work, we propose "tangent images," a spherical image representation that facilitates transferable and scalable 360 degree computer vision. Inspired by techniques in cartography and computer graphics, we render a spherical image to a set of distortion-mitigated, locally-planar image grids tangent to a subdivided icosahedron. By varying the resolution of these grids independently of the subdivision level, we can effectively represent high resolution spherical images while still benefiting from the low-distortion icosahedral spherical approximation. We show that training standard convolutional neural networks on tangent images compares favorably to the many specialized spherical convolutional kernels that have been developed, while also scaling efficiently to handle significantly higher spherical resolutions. Furthermore, because our approach does not require specialized kernels, we show that we can transfer networks trained on perspective images to spherical data without fine-tuning and with limited performance drop-off. Finally, we demonstrate that tangent images can be used to improve the quality of sparse feature detection on spherical images, illustrating its usefulness for traditional computer vision tasks like structure-from-motion and SLAM.
Convolutional Knowledge Tracing: Modeling Individualization in Student Learning Process

With the development of online education systems, a growing number of research works are focusing on Knowledge Tracing (KT), which aims to assess students' changing knowledge state and help them learn knowledge concepts more efficiently. However, only given student learning interactions, most of existing KT methods neglect the individualization of students, i.e., the prior knowledge and learning rates differ from student to student. To this end, in this paper, we propose a novel Convolutional Knowledge Tracing (CKT) method to model individualization in KT. Specifically, for individualized prior knowledge, we measure it from students' historical learning interactions. For individualized learning rates, we design hierarchical convolutional layers to extract them based on continuous learning interactions of students. Extensive experiments demonstrate that CKT could obtain better knowledge tracing results through modeling individualization in learning process. Moreover, CKT can learn meaningful exercise embeddings automatically.
Cross-Modal Pattern-Propagation for RGB-T Tracking

Motivated by our observations on RGB-T data that pattern correlations are high-frequently recurred across modalities also along sequence frames, in this paper, we propose a cross-modal pattern-propagation (CMPP) tracking framework to diffuse instance patterns across RGB-T data on spatial domain as well as temporal domain. To bridge RGB-T modalities, the cross-modal correlations on intra-modal paired pattern-affinities are derived to reveal those latent cues between heterogenous modalities. Through the correlations, the useful patterns may be mutually propagated between RGB-T modalities so as to fulfill inter-modal pattern-propagation. Further, considering the temporal continuity of sequence frames, we adopt the spirit of pattern propagation to dynamic temporal domain, in which long-term historical contexts are adaptively correlated and propagated into the current frame for more effective information inheritance. Extensive experiments demonstrate that the effectiveness of our proposed CMPP, and the new state-of-the-art results are achieved with the significant improvements on two RGB-T object tracking benchmarks.
TITAN: Future Forecast Using Action Priors

We consider the problem of predicting the future trajectory of scene agents from egocentric views obtained from a moving platform. This problem is important in a variety of domains, particularly for autonomous systems making reactive or strategic decisions in navigation. In an attempt to address this problem, we introduce TITAN (Trajectory Inference using Targeted Action priors Network), a new model that incorporates prior positions, actions, and context to forecast future trajectory of agents and future ego-motion. In the absence of an appropriate dataset for this task, we created the TITAN dataset that consists of 700 labeled video-clips (with odometry) captured from a moving vehicle on highly interactive urban traffic scenes in Tokyo. Our dataset includes 50 labels including vehicle states and actions, pedestrian age groups, and targeted pedestrian action attributes that are organized hierarchically corresponding to atomic, simple/complex-contextual, transportive, and communicative actions. To evaluate our model, we conducted extensive experiments on the TITAN dataset, revealing significant performance improvement against baselines and state-of-the-art algorithms. We also report promising results from our Agent Importance Mechanism (AIM), a module which provides insight into assessment of perceived risk by calculating the relative influence of each agent on the future ego-trajectory. The dataset is available at https://usa.honda-ri.com/titan.
Learning to Segment the Tail

Real-world visual recognition requires handling the extreme sample imbalance in large-scale long-tailed data. We propose a “divide&conquer” strategy for the challenging LVIS task: divide the whole data into balanced parts and then apply incremental learning to conquer each one. This derives a novel learning paradigm: class-incremental few-shot learning, which is especially effective for the challenge evolving over time: 1) the class imbalance among the old class knowledge review and 2) the few-shot data in new-class learning. We call our approach Learning to Segment the Tail (LST). In particular, we design an instance-level balanced replay scheme, which is a memory-efficient approximation to balance the instance-level samples from the old-class images. We also propose to use a meta-module for new-class learning, where the module parameters are shared across incremental phases, gaining the learning-to-learn knowledge incrementally, from the data-rich head to the data-poor tail. We empirically show that: at the expense of a little sacrifice of head-class forgetting, we can gain a significant 8.3% AP improvement for the tail classes with less than 10 instances, achieving an overall 2.0% AP boost for the whole 1,230 classes.
PhysGAN: Generating Physical-World-Resilient Adversarial Examples for Autonomous Driving

Although Deep neural networks (DNNs) are being pervasively used in vision-based autonomous driving systems, they are found vulnerable to adversarial attacks where small-magnitude perturbations into the inputs during test time cause dramatic changes to the outputs. While most of the recent attack methods target at digital-world adversarial scenarios, it is unclear how they perform in the physical world, and more importantly, the generated perturbations under such methods would cover a whole driving scene including those fixed background imagery such as the sky, making them inapplicable to physical world implementation. We present PhysGAN, which generates physical-world-resilient adversarial examples for misleading autonomous driving systems in a continuous manner. We show the effectiveness and robustness of PhysGAN via extensive digital- and real-world evaluations. We compare PhysGAN with a set of state-of-the-art baseline methods, which further demonstrate the robustness and efficacy of our approach. We also show that PhysGAN outperforms state-of-the-art baseline methods. To the best of our knowledge, PhysGAN is probably the first technique of generating realistic and physical-world-resilient adversarial examples for attacking common autonomous driving scenarios.
Boundaries and edges rethinking: An end-to-end neural model for overlapping entity relation extraction

Overlapping entity relation extraction has received extensive research attention in recent years. However, existing methods suffer from the limitation of long-distance dependencies between entities, and fail to extract the relations when the overlapping situation is relatively complex. This issue limits the performance of the task. In this paper, we propose an end-to-end neural model for overlapping relation extraction by treating the task as a quintuple prediction problem. The proposed method first constructs the entity graphs by enumerating possible candidate spans, then models the relational graphs between entities via a graph attention model. Experimental results on five benchmark datasets show that the proposed model achieves the current best performance, outperforming previous methods and baseline systems by a large margin. Further analysis shows that our model can effectively capture the long-distance dependencies between entities in a long sentence.
Inflated Episodic Memory With Region Self-Attention for Long-Tailed Visual Recognition

There have been increasing interests in modeling long-tailed data. Unlike artificially collected datasets, long-tailed data are naturally existed in the real-world and thus more realistic. To deal with the class imbalance problem, we introduce an Inflated Episodic Memory (IEM) for long-tailed visual recognition. First, our IEM augments the convolutional neural networks with categorical representative features for rapid learning on tail classes. In traditional few-shot learning, a single prototype is usually leveraged to represent a category. However, long-tailed data has higher intra-class variances. It could be challenging to learn a single prototype for one category. Thus, we introduce IEM to store the most discriminative feature for each category individually. Besides, the memory banks are updated independently, which further decreases the chance of learning skewed classifiers. Second, we introduce a novel region self-attention mechanism for multi-scale spatial feature map encoding. It is beneficial to incorporate more discriminative features to improve generalization on tail classes. We propose to encode local feature maps at multiple scales, and the spatial contextual information should be aggregated at the same time. Equipped with IEM and region self-attention, we achieve state-of-the-art performance on four standard long-tailed image recognition benchmarks. Besides, we validate the effectiveness of IEM on a long-tailed video recognition benchmark, i.e., YouTube-8M.
UAV navigation in high dynamic environments: A deep reinforcement learning approach

Unmanned Aerial Vehicle (UAV) navigation is aimed at guiding a UAV to the desired destinations along a collision-free and efficient path without human interventions, and it plays a crucial role in autonomous missions in harsh environments. The recently emerging Deep Reinforcement Learning (DRL) methods have shown promise for addressing the UAV navigation problem, but most of these methods cannot converge due to the massive amounts of interactive data when a UAV is navigating in high dynamic environments, where there are numerous obstacles moving fast. In this work, we propose an improved DRL-based method to tackle these fundamental limitations. To be specific, we develop a distributed DRL framework to decompose the UAV navigation task into two simpler sub-tasks, each of which is solved through the designed Long Short-Term Memory (LSTM) based DRL network by using only part of the interactive data. Furthermore, a clipped DRL loss function is proposed to closely stack the two sub-solutions into one integral for the UAV navigation problem. Extensive simulation results are provided to corroborate the superiority of the proposed method in terms of the convergence and effectiveness compared with those of the state-of-the-art DRL methods.
Latent space manipulation for high-resolution medical image synthesis via the StyleGAN

This paper explores the potential of the StyleGAN model as an high-resolution image generator for synthetic medical images. The possibility to generate sample patient images of different modalities can be helpful for training deep learning algorithms as e.g. a data augmentation technique. The StyleGAN model was trained on Computed Tomography (CT) and T2- weighted Magnetic Resonance (MR) images from 100 patients with pelvic malignancies. The resulting model was investigated with regards to three features: Image Modality, Sex, and Longitudinal Slice Position. Further, the style transfer feature of the StyleGAN was used to move images between the modalities. The root-mean-squard error (RMSE) and the Mean Absolute Error (MAE) were used to quantify errors for MR and CT, respectively. We demonstrate how these features can be transformed by manipulating the latent style vectors, and attempt to quantify how the errors change as we move through the latent style space. The best results were achieved by using the style transfer feature of the StyleGAN (58.7 HU MAE for MR to CT and 0.339 RMSE for CT to MR). Slices below and above an initial central slice can be predicted with an error below 75 HU MAE and 0.3 RMSE within 4 cm for CT and MR, respectively. The StyleGAN is a promising model to use for generating synthetic medical images for MR and CT modalities as well as for 3D volumes.
A new image encryption scheme based on coupling map lattices with mixed multi-chaos

As a kind of spatiotemporal chaos, coupled map lattice (CML) is widely applied into image encryption because of its advantages of more complex dynamical behavior and lower computational overhead. Firstly, this paper proposed a novel spatiotemporal chaos model (MCML) by mixing Logistic, Sine and Tent maps into CML map together. Beyond that, we also change the structure of CML and the coupling method in different lattices. Bifurcation diagram, Lyapunov exponents and NIST test are employed to measure the chaotic behaviors of the MCML system. Secondly, by applying MCML chaos, we design a new key binding and distribution rule, the improved diffusion scheme to encrypt image. Furthermore, the novel bit Z-scan scrambling method also be used to enhance the security of the encryption scheme. Finally, a large number of experimental results prove that our proposed scheme is suitable for image encryption and has high security against common attacks.
Social media as a platform in academic library marketing: A comparative study

Social media like Facebook have become popular tools for different organizations like libraries in marketing practice. To build relationships with library users, libraries hope social media can engage its user communities actively with their collections, services, and activities. This paper aims at evaluating the effectiveness of using social media as a platform in marketing through a questionnaire on the Facebook page of the University of Hong Kong Libraries (HKUL), comparing the perspectives of students and faculty members. Both the current situation of HKUL's Facebook page and the reasons affecting users' interest and participation in the page are evaluated, in order to suggest better strategies for the library to deal with the needs of library users in the future. Other university libraries can also gain new insights from the study. This research has the following key findings: 1. The marketing practices of HKUL's Facebook page generally did not receive adequate attention and reactions from users; 2. Students were more engaged than faculty members in HKUL's Facebook page, as students use more varieties of library services than faculty members; 3. User needs, social media content, and interactions generally affected user acceptance of the library's Facebook marketing.
Fast color image encryption scheme based on 3D orthogonal Latin squares and matching matrix

To realize real-time image encryption, a fast color image encryption scheme by combining 3D orthogonal Latin squares (3D-OLSs) with matching matrix is proposed. The 3D-OLSs represent that each plane of two matrices must be Latin square and the corresponding planes of the two matrices must satisfy orthogonality. The matching matrix is to produce a matrix orthogonal with the 3D Latin square. In the permutation process, a new 3D permutation method with 3D-OLSs and matching matrix is devised. The proposed scheme could save encryption time to a certain degree, since the orthogonal Latin squares are defined over integers directly. In the diffusion process, to solve the diffuse problem between two planes in the 3D matrix, some matrices of the diffusion process are changed with three variables. Experimental results and security analyses show that the proposed color image encryption scheme has high security, fast speed and could resist various common attacks.
Considerably Improving Clustering Algorithms Using UMAP Dimensionality Reduction Technique: A Comparative Study

Dimensionality reduction is widely used in machine learning and big data analytics since it helps to analyze and to visualize large, high-dimensional datasets. In particular, it can considerably help to perform tasks like data clustering and classification. Recently, embedding methods have emerged as a promising direction for improving clustering accuracy. They can preserve the local structure and simultaneously reveal the global structure of data, thereby reasonably improving clustering performance. In this paper, we investigate how to improve the performance of several clustering algorithms using one of the most successful embedding techniques: Uniform Manifold Approximation and Projection or UMAP. This technique has recently been proposed as a manifold learning technique for dimensionality reduction. It is based on Riemannian geometry and algebraic topology. Our main hypothesis is that UMAP would permit to find the best clusterable embedding manifold, and therefore, we applied it as a preprocessing step before performing clustering. We compare the results of many well-known clustering algorithms such ask-means, HDBSCAN, GMM and Agglomerative Hierarchical Clustering when they operate on the low-dimension feature space yielded by UMAP. A series of experiments on several image datasets demonstrate that the proposed method allows each of the clustering algorithms studied to improve its performance on each dataset considered. Based on Accuracy measure, the improvement can reach a remarkable rate of 60%.
Kidney segmentation from computed tomography images using deep neural network

The precise segmentation of kidneys and kidney tumors can help medical specialists to diagnose diseases and improve treatment planning, which is highly required in clinical practice. Manual segmentation of the kidneys is extremely time-consuming and prone to variability between different specialists due to their heterogeneity. Because of this hard work, computational techniques, such as deep convolutional neural networks, have become popular in kidney segmentation tasks to assist in the early diagnosis of kidney tumors. In this study, we propose an automatic method to delimit the kidneys in computed tomography (CT) images using image processing techniques and deep convolutional neural networks (CNNs) to minimize false positives. The proposed method has four main steps: (1) acquisition of the KiTS19 dataset, (2) scope reduction using AlexNet, (3) initial segmentation using U-Net 2D, and (4) false positive reduction using image processing to maintain the largest elements (kidneys). The proposed method was evaluated in 210 CTs from the KiTS19 database and obtained the best result with an average Dice coefficient of 96.33%, an average Jaccard index of 93.02%, an average sensitivity of 97.42%, an average specificity of 99.94% and an average accuracy of 99.92%. In the KiTS19 challenge, it presented an average Dice coefficient of 93.03%. In our method, we demonstrated that the kidney segmentation problem in CT can be solved efficiently using deep neural networks to define the scope of the problem and segment the kidneys with high precision and with the use of image processing techniques to reduce false positives.
A Multi-Neural Network Acceleration Architecture

A cost-effective multi-tenant neural network execution is becoming one of the most important design goals for modern neural network accelerators. For example, as emerging AI services consist of many heterogeneous neural network executions, a cloud provider wants to serve a large number of clients using a single AI accelerator for improving its cost effectiveness. Therefore, an ideal next-generation neural network accelerator should support a simultaneous multi-neural network execution, while fully utilizing its hardware resources. However, existing accelerators which are optimized for a single neural network execution can suffer from severe resource underutilization when running multiple neural networks, mainly due to the load imbalance between computation and memory-access tasks from different neural networks.In this paper, we propose AI-MultiTasking (AI-MT), a novel accelerator architecture which enables a cost-effective, high-performance multi-neural network execution. The key idea of AI-MT is to fully utilize the accelerator’s computation resources and memory bandwidth by matching compute- and memory-intensive tasks from different networks and executing them in parallel. However, it is highly challenging to find and schedule the best load-matching tasks from different neural networks during runtime, without significantly increasing the size of on-chip memory. To overcome the challenges, AI-MT first creates fine-grain tasks at compile time by dividing each layer into multiple identical sub-layers. During runtime, AI-MT dynamically applies three sub-layer scheduling methods: memory block prefetching and compute block merging for the best resource load matching, and memory block eviction for the minimum on-chip memory footprint. Our evaluations using MLPerf benchmarks show that AI-MT achieves up to 1.57x speedup over the baseline scheduling method.
Multi-Path Deep CNNs for Fine-Grained Car Recognition

Along with the growing demands of intelligent traffic system, how to recognize the category information of a car from surveillance cameras has been an important task. Fine-grained car recognition is facing challenges mainly due to similar appearance of inter-class car images. Moreover, unlike other forms of object recognition, there are a large quantity of car models, that most other objects do not have, which causes fine-grained car recognition to be of vital importance but challenging. Recent research focuses on training a deep convolution neural network (DCNN) on a large dataset. Typically these methods largely rely on the holistic appearance of cars and may fail to train an optimal DCNN to comprehend information from multiple parts of a car. To address this issue, we carefully measure the effectiveness of different parts of a car and highlight the importance of car fronts for fine-grained car recognition. Then, considering multiple important parts from cars, we propose a novel multi-path DCNN model which is equipped with a 3-branch deep convolutional network to better exploit holistic as well as part information for fine-grained car recognition. Multiple parts from cars provide complementary information to boost performance. To further facilitate research on fine-grained car recognition, we also collect a large-scale dataset named “Multiple Parts from Cars” (MPF-Cars) which contains category annotation as well as car part information. We evaluate our proposed multi-path DCNN on MPF-Cars and another benchmark dataset CarFlag-563. Experiments demonstrate the effectiveness of our proposed method.