OpenRaiser/Intern-Atlas huggingface dataset:
Simulating Personal Food Consumption Patterns using a Modified Markov Chain
Venue: arXiv cs.CV 2022
Year: 2022

Food image classification serves as the foundation of image-based dietary assessment to predict food categories. Since there are many different food classes in real life, conventional models cannot achieve sufficiently high accuracy. Personalized classifiers aim to largely improve the accuracy of food image classification for each individual. However, a lack of public personal food consumption data proves to be a challenge for training such models. To address this issue, we propose a novel framework to simulate personal food consumption data patterns, leveraging the use of a modified Markov chain model and self-supervised learning. Our method is capable of creating an accurate future data pattern from a limited amount of initial data, and our simulated data patterns can be closely correlated with the initial data pattern. Furthermore, we use Dynamic Time Warping distance and Kullback-Leibler divergence as metrics to evaluate the effectiveness of our method on the public Food-101 dataset. Our experimental results demonstrate promising performance compared with random simulation and the original Markov chain method.
MAFNet: A Multi-Attention Fusion Network for RGB-T Crowd Counting
Venue: arXiv cs.CV 2022
Year: 2022

RGB-Thermal (RGB-T) crowd counting is a challenging task, which uses thermal images as complementary information to RGB images to deal with the decreased performance of unimodal RGB-based methods in scenes with low-illumination or similar backgrounds. Most existing methods propose well-designed structures for cross-modal fusion in RGB-T crowd counting. However, these methods have difficulty in encoding cross-modal contextual semantic information in RGB-T image pairs. Considering the aforementioned problem, we propose a two-stream RGB-T crowd counting network called Multi-Attention Fusion Network (MAFNet), which aims to fully capture long-range contextual information from the RGB and thermal modalities based on the attention mechanism. Specifically, in the encoder part, a Multi-Attention Fusion (MAF) module is embedded into different stages of the two modality-specific branches for cross-modal fusion at the global level. In addition, a Multi-modal Multi-scale Aggregation (MMA) regression head is introduced to make full use of the multi-scale and contextual information across modalities to generate high-quality crowd density maps. Extensive experiments on two popular datasets show that the proposed MAFNet is effective for RGB-T crowd counting and achieves the state-of-the-art performance.
DisenHCN: Disentangled Hypergraph Convolutional Networks for Spatiotemporal Activity Prediction
Venue: arXiv cs.LG 2022
Year: 2022

Spatiotemporal activity prediction, aiming to predict user activities at a specific location and time, is crucial for applications like urban planning and mobile advertising. Existing solutions based on tensor decomposition or graph embedding suffer from the following two major limitations: 1) ignoring the fine-grained similarities of user preferences; 2) user's modeling is entangled. In this work, we propose a hypergraph neural network model called DisenHCN to bridge the above gaps. In particular, we first unify the fine-grained user similarity and the complex matching between user preferences and spatiotemporal activity into a heterogeneous hypergraph. We then disentangle the user representations into different aspects (location-aware, time-aware, and activity-aware) and aggregate corresponding aspect's features on the constructed hypergraph, capturing high-order relations from different aspects and disentangles the impact of each aspect for final prediction. Extensive experiments show that our DisenHCN outperforms the state-of-the-art methods by 14.23% to 18.10% on four real-world datasets. Further studies also convincingly verify the rationality of each component in our DisenHCN.
Semi-Supervised Video Inpainting with Cycle Consistency Constraints
Venue: arXiv cs.CV 2022
Year: 2022

Deep learning-based video inpainting has yielded promising results and gained increasing attention from researchers. Generally, these methods usually assume that the corrupted region masks of each frame are known and easily obtained. However, the annotation of these masks are labor-intensive and expensive, which limits the practical application of current methods. Therefore, we expect to relax this assumption by defining a new semi-supervised inpainting setting, making the networks have the ability of completing the corrupted regions of the whole video using the annotated mask of only one frame. Specifically, in this work, we propose an end-to-end trainable framework consisting of completion network and mask prediction network, which are designed to generate corrupted contents of the current frame using the known mask and decide the regions to be filled of the next frame, respectively. Besides, we introduce a cycle consistency loss to regularize the training parameters of these two networks. In this way, the completion network and the mask prediction network can constrain each other, and hence the overall performance of the trained model can be maximized. Furthermore, due to the natural existence of prior knowledge (e.g., corrupted contents and clear borders), current video inpainting datasets are not suitable in the context of semi-supervised video inpainting. Thus, we create a new dataset by simulating the corrupted video of real-world scenarios. Extensive experimental results are reported to demonstrate the superiority of our model in the video inpainting task. Remarkably, although our model is trained in a semi-supervised manner, it can achieve comparable performance as fully-supervised methods.
Generic Go to Go: Dictionary-Passing, Monomorphisation, and Hybrid
Venue: arXiv cs.PL 2022
Year: 2022

Go is a popular statically-typed industrial programming language. To aid the type safe reuse of code, the recent Go release (Go 1.18) published on 15th March 2022 includes bounded parametric polymorphism via generic types. Go 1.18 implements generic types using combination of monomorphisation and call-graph based dictionary-passing called hybrid. This hybrid approach can be viewed as an optimised form of monomorphisation that statically generates specialised methods and types based on possible instantiations. A monolithic dictionary supplements information lost during monomorphisation, and it is structured according to the program's call graph. Unfortunately, the hybrid approach still suffers from code bloat, poor compilation speed, and limited code coverage. In this paper we propose and formalise a new non-specialising call-site based dictionary-passing translation. Our call-site based translation creates individual dictionaries for each type parameter, with dictionary construction occurring in place of instantiation, overcoming the limitations of hybrid. We prove it correct using a novel and general bisimulation up to technique. To better understand how different generics translations approaches work in practice, we benchmark five translators, Go 1.18, two existing monomorphisation translators, our dictionary-passing translator, and erasure translator. Our findings reveal several suggestions for improvements for Go 1.18 -- specifically how to overcome the expressiveness limitations of generic Go, and improve compile time and compiled code size performance of Go 1.18.
An Improved Greedy Algorithm for Stochastic Online Scheduling on Unrelated Machines
Venue: arXiv cs.DS 2022
Year: 2022

Most practical scheduling applications involve some uncertainty about the arriving times and lengths of the jobs. Stochastic online scheduling is a well-established model capturing this. Here the arrivals occur online, while the processing times are random. For this model, Gupta, Moseley, Uetz, and Xie recently devised an efficient policy for non-preemptive scheduling on unrelated machines with the objective to minimize the expected total weighted completion time. We improve upon this policy by adroitly combining greedy job assignment with $α_j$-point scheduling on each machine. In this way we obtain a $(3+\sqrt 5)(2+Δ)$-competitive deterministic and an $(8+4Δ)$-competitive randomized stochastic online scheduling policy, where $Δ$ is an upper bound on the squared coefficients of variation of the processing times. We also give constant performance guarantees for these policies within the class of all fixed-assignment policies. The $α_j$-point scheduling on a single machine can be enhanced when the upper bound $Δ$ is known a priori or the processing times are known to be $δ$-NBUE for some $δ\ge 1$. This implies improved competitive ratios for unrelated machines but may also be of independent interest.
An Updated Database of $\mathbb{Z}_4$ Codes
Venue: arXiv cs.IT 2022
Year: 2022

Research on codes over finite rings has intensified since the discovery in 1994 of the fact that some best binary non-linear codes can be obtained as images of $\mathbb{Z}_4$-linear codes. Codes over many different finite rings has been a subject of much research in coding theory after this discovery. Many of these rings are extensions of $\mathbb{Z}_4$. As a result, an online database of $\mathbb{Z}_4$ was created in 2008. The URL of the original database on $\mathbb{Z}_4$ codes has recently changed. The purpose of this paper is to introduce the new, updated database of $\mathbb{Z}_4$ codes. We have made major updates to the database by adding 8701 new linear codes over $\mathbb{Z}_4$. These codes have been found through exhaustive computer searches on cyclic codes and by an implementation of the ASR search algorithm that has been remarkably fruitful to obtain new linear codes from the class of quasi-cyclic (QC) and quasi-twisted (QT) codes over finite fields. We made modifications to the ASR algorithm to make it work over $\mathbb{Z}_4$. The initial database contained few codes that were not free. We have added a large number of non-free codes. In fact, of the 8701 codes we have added, 7631 of them are non-free.
Forgetting Fast in Recommender Systems
Venue: arXiv cs.IR 2022
Year: 2022

Users of a recommender system may want part of their data being deleted, not only from the data repository but also from the underlying machine learning model, for privacy or utility reasons. Such right-to-be-forgotten requests could be fulfilled by simply retraining the recommendation model from scratch, but that would be too slow and too expensive in practice. In this paper, we investigate fast machine unlearning techniques for recommender systems that can remove the effect of a small amount of training data from the recommendation model without incurring the full cost of retraining. A natural idea to speed this process up is to fine-tune the current recommendation model on the remaining training data instead of starting from a random initialization. This warm-start strategy indeed works for neural recommendation models using standard 1st-order neural network optimizers (like AdamW). However, we have found that even greater acceleration could be achieved by employing 2nd-order (Newton or quasi-Newton) optimization methods instead. To overcome the prohibitively high computational cost of 2nd-order optimizers, we propose a new recommendation unlearning approach AltEraser which divides the optimization problem of unlearning into many small tractable sub-problems. Extensive experiments on three real-world recommendation datasets show promising results of AltEraser in terms of consistency (forgetting thoroughness), accuracy (recommendation effectiveness), and efficiency (unlearning speed). To our knowledge, this work represents the first attempt at fast approximate machine unlearning for state-of-the-art neural recommendation models.
Conformal Navigation Transformations with Application to Robot Navigation in Complex Workspaces
Venue: arXiv cs.RO 2022
Year: 2022

Navigation functions provide both path and motion planning, which can be used to ensure obstacle avoidance and convergence in the sphere world. When dealing with complex and realistic scenarios, constructing a transformation to the sphere world is essential and, at the same time, challenging. This work proposes a novel transformation termed the conformal navigation transformation to achieve collision-free navigation of a robot in a workspace populated with obstacles of arbitrary shapes. The properties of the conformal navigation transformation, including uniqueness, invariance of navigation properties, and no angular deformation, are investigated, which contribute to the solution of the robot navigation problem in complex environments. Based on navigation functions and the proposed transformation, feedback controllers are derived for the automatic guidance and motion control of kinematic and dynamic mobile robots. Moreover, an iterative method is proposed to construct the conformal navigation transformation in a multiply-connected workspace, which transforms the multiply-connected problem into multiple simply-connected problems to achieve fast convergence. In addition to the analytic guarantees, simulation studies verify the effectiveness of the proposed methodology in workspaces with non-trivial obstacles.
CoShNet: A Hybrid Complex Valued Neural Network using Shearlets
Venue: arXiv cs.CV 2022
Year: 2022

In a hybrid neural network, the expensive convolutional layers are replaced by a non-trainable fixed transform with a great reduction in parameters. In previous works, good results were obtained by replacing the convolutions with wavelets. However, wavelet based hybrid network inherited wavelet's lack of vanishing moments along curves and its axis-bias. We propose to use Shearlets with its robust support for important image features like edges, ridges and blobs. The resulting network is called Complex Shearlets Network (CoShNet). It was tested on Fashion-MNIST against ResNet-50 and Resnet-18, obtaining 92.2% versus 90.7% and 91.8% respectively. The proposed network has 49.9k parameters versus ResNet-18 with 11.18m and use 52 times fewer FLOPs. Finally, we trained in under 20 epochs versus 200 epochs required by ResNet and do not need any hyperparameter tuning nor regularization. Code: https://github.com/Ujjawal-K-Panchal/coshnet
Global Priors Guided Modulation Network for Joint Super-Resolution and Inverse Tone-Mapping
Venue: arXiv cs.CV 2022
Year: 2022

Joint super-resolution and inverse tone-mapping (SR-ITM) aims to enhance the visual quality of videos that have quality deficiencies in resolution and dynamic range. This problem arises when using 4K high dynamic range (HDR) TVs to watch a low-resolution standard dynamic range (LR SDR) video. Previous methods that rely on learning local information typically cannot do well in preserving color conformity and long-range structural similarity, resulting in unnatural color transition and texture artifacts. In order to tackle these challenges, we propose a global priors guided modulation network (GPGMNet) for joint SR-ITM. In particular, we design a global priors extraction module (GPEM) to extract color conformity prior and structural similarity prior that are beneficial for ITM and SR tasks, respectively. To further exploit the global priors and preserve spatial information, we devise multiple global priors guided spatial-wise modulation blocks (GSMBs) with a few parameters for intermediate feature modulation, in which the modulation parameters are generated by the shared global priors and the spatial features map from the spatial pyramid convolution block (SPCB). With these elaborate designs, the GPGMNet can achieve higher visual quality with lower computational complexity. Extensive experiments demonstrate that our proposed GPGMNet is superior to the state-of-the-art methods. Specifically, our proposed model exceeds the state-of-the-art by 0.64 dB in PSNR, with 69$\%$ fewer parameters and 3.1$\times$ speedup. The code will be released soon.
The SVD of Convolutional Weights: A CNN Interpretability Framework
Venue: arXiv cs.CV 2022
Year: 2022

Deep neural networks used for image classification often use convolutional filters to extract distinguishing features before passing them to a linear classifier. Most interpretability literature focuses on providing semantic meaning to convolutional filters to explain a model's reasoning process and confirm its use of relevant information from the input domain. Fully connected layers can be studied by decomposing their weight matrices using a singular value decomposition, in effect studying the correlations between the rows in each matrix to discover the dynamics of the map. In this work we define a singular value decomposition for the weight tensor of a convolutional layer, which provides an analogous understanding of the correlations between filters, exposing the dynamics of the convolutional map. We validate our definition using recent results in random matrix theory. By applying the decomposition across the linear layers of an image classification network we suggest a framework against which interpretability methods might be applied using hypergraphs to model class separation. Rather than looking to the activations to explain the network, we use the singular vectors with the greatest corresponding singular values for each linear layer to identify those features most important to the network. We illustrate our approach with examples and introduce the DeepDataProfiler library, the analysis tool used for this study.
Model Generalization: A Sharpness Aware Optimization Perspective
Venue: arXiv cs.LG 2022
Year: 2022

Sharpness-Aware Minimization (SAM) and adaptive sharpness-aware minimization (ASAM) aim to improve the model generalization. And in this project, we proposed three experiments to valid their generalization from the sharpness aware perspective. And our experiments show that sharpness aware-based optimization techniques could help to provide models with strong generalization ability. Our experiments also show that ASAM could improve the generalization performance on un-normalized data, but further research is needed to confirm this.
Visual Localization via Few-Shot Scene Region Classification
Venue: arXiv cs.CV 2022
Year: 2022

Visual (re)localization addresses the problem of estimating the 6-DoF (Degree of Freedom) camera pose of a query image captured in a known scene, which is a key building block of many computer vision and robotics applications. Recent advances in structure-based localization solve this problem by memorizing the mapping from image pixels to scene coordinates with neural networks to build 2D-3D correspondences for camera pose optimization. However, such memorization requires training by amounts of posed images in each scene, which is heavy and inefficient. On the contrary, few-shot images are usually sufficient to cover the main regions of a scene for a human operator to perform visual localization. In this paper, we propose a scene region classification approach to achieve fast and effective scene memorization with few-shot images. Our insight is leveraging a) pre-learned feature extractor, b) scene region classifier, and c) meta-learning strategy to accelerate training while mitigating overfitting. We evaluate our method on both indoor and outdoor benchmarks. The experiments validate the effectiveness of our method in the few-shot setting, and the training time is significantly reduced to only a few minutes. Code available at: \url{https://github.com/siyandong/SRC}
Novel Ordering-based Approaches for Causal Structure Learning in the Presence of Unobserved Variables
Venue: arXiv cs.LG 2022
Year: 2022

We propose ordering-based approaches for learning the maximal ancestral graph (MAG) of a structural equation model (SEM) up to its Markov equivalence class (MEC) in the presence of unobserved variables. Existing ordering-based methods in the literature recover a graph through learning a causal order (c-order). We advocate for a novel order called removable order (r-order) as they are advantageous over c-orders for structure learning. This is because r-orders are the minimizers of an appropriately defined optimization problem that could be either solved exactly (using a reinforcement learning approach) or approximately (using a hill-climbing search). Moreover, the r-orders (unlike c-orders) are invariant among all the graphs in a MEC and include c-orders as a subset. Given that set of r-orders is often significantly larger than the set of c-orders, it is easier for the optimization problem to find an r-order instead of a c-order. We evaluate the performance and the scalability of our proposed approaches on both real-world and randomly generated networks.
Multirotor Planning in Dynamic Environments using Temporal Safe Corridors
Venue: arXiv cs.RO 2022
Year: 2022

In this paper, we propose a new method for multirotor planning in dynamic environments. The environment is represented as a temporal occupancy grid which gives the current as well as the future/predicted state of all the obstacles. The method builds on previous works in Safe Corridor generation and multirotor planning to avoid moving and static obstacles. It first generates a global path to the goal that doesn't take into account the dynamic aspect of the environment. We then use temporal Safe Corridors to generate safe spaces that the robot can be in at discrete instants in the future. Finally we use the temporal Safe Corridors in an optimization formulation that accounts for the multirotor dynamics as well as all the obstacles to generate the trajectory that will be executed by the multirotor's controller. We show the performance of our method in simulations.
Any strongly controllable group system or group shift or any linear block code is isomorphic to a generator group
Venue: arXiv cs.IT 2022
Year: 2022

Consider any sequence of finite groups $A^t$, where $t$ takes values in an integer index set $\mathbf{Z}$. A group system $A$ is a set of sequences with components in $A^t$ that forms a group under componentwise addition in $A^t$, for each $t\in\mathbf{Z}$. As shown previously, any strongly controllable complete group system $A$ can be decomposed into generators. We study permutations of the generators when sequences in the group system are multiplied. We show that any strongly controllable complete group system $A$ is isomorphic to a generator group $({\mathcal{U}},\circ)$. The set ${\mathcal{U}}$ is a set of tensors, a double Cartesian product space of sets $G_k^t$, with indices $k$, for $0\le k\le\ell$, and time $t$, for $t\in\mathbf{Z}$. $G_k^t$ is a set of unique generator labels for the generators in $A$ with nontrivial span for the time interval $[t,t+k]$. We show the generator group contains a unique elementary system, an infinite collection of elementary groups, one for each $k$ and $t$, defined on small subsets of ${\mathcal{U}}$, in the shape of triangles, which form a tile like structure over ${\mathcal{U}}$. There is a homomorphism from each elementary group to any elementary group defined on smaller tiles of the former group. The group system $A$ may be constructed from either the generator group or elementary system. These results have application to linear block codes, any algebraic system that contains a linear block code, group shifts, and harmonic theory in mathematics, and systems theory, coding theory, control theory, and related fields in engineering.