Miracle-dz/UniFAR-FTUs huggingface dataset:
Graph Neural Networks for User Identity Linkage

The increasing popularity and diversity of social media sites has encouraged more and more people to participate in multiple online social networks to enjoy their services. Each user may create a user identity to represent his or her unique public figure in every social network. User identity linkage across online social networks is an emerging task and has attracted increasing attention, which could potentially impact various domains such as recommendations and link predictions. The majority of existing work focuses on mining network proximity or user profile data for discovering user identity linkages. With the recent advancements in graph neural networks (GNNs), it provides great potential to advance user identity linkage since users are connected in social graphs, and learning latent factors of users and items is the key. However, predicting user identity linkages based on GNNs faces challenges. For example, the user social graphs encode both \textit{local} structure such as users' neighborhood signals, and \textit{global} structure with community properties. To address these challenges simultaneously, in this paper, we present a novel graph neural network framework ({\m}) for user identity linkage. In particular, we provide a principled approach to jointly capture local and global information in the user-user social graph and propose the framework {\m}, which jointly learning user representations for user identity linkage. Extensive experiments on real-world datasets demonstrate the effectiveness of the proposed framework.
Spectral Method for Phase Retrieval: An Expectation Propagation Perspective

Phase retrieval refers to the problem of recovering a signal <inline-formula> <tex-math notation="LaTeX">$ {x}_{\star }\in \mathbb {C}^{n}$ </tex-math></inline-formula> from its phaseless measurements <inline-formula> <tex-math notation="LaTeX">$\text {y}_{\text {i}}=| {a}_{i}^{ \mathsf {H}} {x}_{\star }|$ </tex-math></inline-formula>, where <inline-formula> <tex-math notation="LaTeX">$\{ {a}_{\text {i}}\}_{\text {i}=1}^{ {m}}$ </tex-math></inline-formula> are the measurement vectors. Spectral method is widely used for initialization in many phase retrieval algorithms. The quality of spectral initialization can have a major impact on the overall algorithm. In this paper, we focus on the model where <inline-formula> <tex-math notation="LaTeX">$ {A}=[ {a}_{1},\ldots, {a}_{ {m}}]^{ \mathsf {H}}$ </tex-math></inline-formula> has orthonormal columns, and study the spectral initialization under the asymptotic setting <inline-formula> <tex-math notation="LaTeX">$ {m}, {n}\to \infty $ </tex-math></inline-formula> with <inline-formula> <tex-math notation="LaTeX">$ {m}/ {n}\to \delta \in (1,\infty)$ </tex-math></inline-formula>. We use the expectation propagation framework to characterize the performance of spectral initialization for Haar distributed matrices. Our numerical results confirm that the predictions of the EP method are accurate for not-only Haar distributed matrices, but also for realistic Fourier based models (e.g. the coded diffraction model). The main findings of this paper are the following: 1) There exists a threshold on <inline-formula> <tex-math notation="LaTeX">$\delta $ </tex-math></inline-formula> (denoted as <inline-formula> <tex-math notation="LaTeX">$\delta _{ \mathrm {weak}}$ </tex-math></inline-formula>) below which the spectral method cannot produce a meaningful estimate. We show that <inline-formula> <tex-math notation="LaTeX">$\delta _{ \mathrm {weak}}=2$ </tex-math></inline-formula> for the column-orthonormal model. In contrast, previous results by Mondelli and Montanari show that <inline-formula> <tex-math notation="LaTeX">$\delta _{ \mathrm {weak}}=1$ </tex-math></inline-formula> for the i.i. d. Gaussian model. 2) The optimal design for the spectral method coincides with that for the i.i. d. Gaussian model, where the latter was recently introduced by Luo, Alghamdi and Lu.
Dixit: Interactive Visual Storytelling via Term Manipulation

In this paper, we introduce Dixit, an interactive visual storytelling system that the user interacts with iteratively to compose a short story for a photo sequence. The user initiates the process by uploading a sequence of photos. Dixit first extracts text terms from each photo which describe the objects (e.g., boy, bike) or actions (e.g., sleep) in the photo, and then allows the user to add new terms or remove existing terms. Dixit then generates a short story based on these terms. Behind the scenes, Dixit uses an LSTM-based model trained on image caption data and FrameNet to distill terms from each image, and utilizes a transformer decoder to compose a context-coherent story. Users change images or terms iteratively with Dixit to create the most ideal story. Dixit also allows users to manually edit and rate stories. The proposed procedure opens up possibilities for interpretable and controllable visual storytelling, allowing users to understand the story formation rationale and to intervene in the generation process.
Direct Feedback Alignment With Sparse Connections for Local Learning

Recent advances in deep neural networks (DNNs) owe their success to training algorithms that use backpropagation and gradient-descent. Backpropagation, while highly effective on von Neumann architectures, becomes inefficient when scaling to large networks. Commonly referred to as the weight transport problem, each neuron's dependence on the weights and errors located deeper in the network require exhaustive data movement which presents a key problem in enhancing the performance and energy-efficiency of machine-learning hardware. In this work, we propose a bio-plausible alternative to backpropagation drawing from advances in feedback alignment algorithms in which the error computation at a single synapse reduces to the product of three scalar values. Using a sparse feedback matrix, we show that a neuron needs only a fraction of the information previously used by the feedback alignment algorithms. Consequently, memory and compute can be partitioned and distributed whichever way produces the most efficient forward pass so long as a single error can be delivered to each neuron. We evaluate our algorithm using standard datasets, including ImageNet, to address the concern of scaling to challenging problems. Our results show orders of magnitude improvement in data movement and 2× improvement in multiply-and-accumulate operations over backpropagation. Like previous work, we observe that any variant of feedback alignment suffers significant losses in classification accuracy on deep convolutional neural networks. By transferring trained convolutional layers and training the fully connected layers using direct feedback alignment, we demonstrate that direct feedback alignment can obtain results competitive with backpropagation. Furthermore, we observe that using an extremely sparse feedback matrix, rather than a dense one, results in a small accuracy drop while yielding hardware advantages. All the code and results are available under https://github.com/bcrafton/ssdfa.
Image captioning with weakly-supervised attention penalty

Stories are essential for genealogy research since they can help build emotional connections with people. A lot of family stories are reserved in historical photos and albums. Recent development on image captioning models makes it feasible to "tell stories" for photos automatically. The attention mechanism has been widely adopted in many state-of-the-art encoder-decoder based image captioning models, since it can bridge the gap between the visual part and the language part. Most existing captioning models implicitly trained attention modules with word-likelihood loss. Meanwhile, lots of studies have investigated intrinsic attentions for visual models using gradient-based approaches. Ideally, attention maps predicted by captioning models should be consistent with intrinsic attentions from visual models for any given visual concept. However, no work has been done to align implicitly learned attention maps with intrinsic visual attentions. In this paper, we proposed a novel model that measured consistency between captioning predicted attentions and intrinsic visual attentions. This alignment loss allows explicit attention correction without using any expensive bounding box annotations. We developed and evaluated our model on COCO dataset as well as a genealogical dataset from this http URL Operations Inc., which contains billions of historical photos. The proposed model achieved better performances on all commonly used language evaluation metrics for both datasets.
Negative Training for Neural Dialogue Response Generation

Although deep learning models have brought tremendous advancements to the field of open-domain dialogue response generation, recent research results have revealed that the trained models have undesirable generation behaviors, such as malicious responses and generic (boring) responses. In this work, we propose a framework named “Negative Training” to minimize such behaviors. Given a trained model, the framework will first find generated samples that exhibit the undesirable behavior, and then use them to feed negative training signals for fine-tuning the model. Our experiments show that negative training can significantly reduce the hit rate of malicious responses, or discourage frequent responses and improve response diversity.
PatchNet: A Tool for Deep Patch Classification

This work proposes PatchNet, an automated tool based on hierarchical deep learning for classifying patches by extracting features from commit messages and code changes. PatchNet contains a deep hierarchical structure that mirrors the hierarchical and sequential structure of a code change, differentiating it from the existing deep learning models on source code. PatchNet provides several options allowing users to select parameters for the training process. The tool has been validated in the context of automatic identification of stable-relevant patches in the Linux kernel and is potentially applicable to automate other software engineering tasks that can be formulated as patch classification problems. Our video demonstrating PatchNet and PatchNet implementation are publicly available at https: //goo.gl/CZjG6X and https://github.com/hvdthong/PatchNetTool respectively.
RINS-W: Robust Inertial Navigation System on Wheels

This paper proposes a real-time approach for long-term inertial navigation based only on an Inertial Measurement Unit (IMU) for self-localizing wheeled robots. The approach builds upon two components: 1) a robust detector that uses recurrent deep neural networks to dynamically detect a variety of situations of interest, such as zero velocity or no lateral slip; and 2) a state-of-the-art Kalman filter which incorporates this knowledge as pseudo-measurements for localization. Evaluations on a publicly available car dataset demonstrates that the proposed scheme may achieve a final distance error of 20 m for a 21 km long trajectory of a vehicle driving for over an hour, equipped with an IMU of moderate precision (the gyro drift rate is 10 deg/h). To our knowledge, this is the first paper which combines sophisticated deep learning techniques with state-of the-art filtering methods for pure inertial navigation on wheeled vehicles and as such opens up for novel data-driven inertial navigation techniques. Moreover, albeit taylored for IMU-only based localization, our method may be used as a component for self-localization of wheeled robots equipped with a more complete sensor suite.
IP3R Ca2+ release shapes the cytosolic Ca2+ transient for hypertrophic signalling in cardiomyocytes.

Calcium plays a central role in mediating both contractile function and hypertrophicsignalling in cardiomyocytes. While L-type Ca2+ channels (LCCs) trigger beat-to-beat re-lease of calcium through ryanodine receptors (RyRs) for cellular contraction, inositol 1,4,5-triphosphate receptors (IP3Rs) have been implicated in releasing Ca2+ to modify beat-to-beat calcium to concurrently signal translocation of transcription factor NFAT into the nucleus to initiate hypertrophy. Details of both the generation and delivery of this translocation signal are unclear however. A recent study by Hannanta-anan and Chow has indicated that NFAT integrates the calcium signal over time. Their experiments revealed that an increase in the duration of elevated cytosolic calcium within each period of oscillation (increase in the duty cycle) increases NFAT-mediated gene transcription. However, it is not clear whether this mechanism exists in cardiac cells, or is even the mechanism underlying NFAT activation. Here we present a mathematical model of functional interactions between RyR and IP3R channels which demonstrates that IP3-mediated Ca2+release in the cytosol is indeed capable of increasing the duty cycle of the calcium signal in the cardiomyocyte. Through comparison with published experimental results we demonstrate the importance of appropriately representing IP3R gating, verify that our model is capable of reproducing known disease phenotypes in the calcium transient through alteration of channel properties, and examine how sensitive the cardiac cell is to IP3. A key result of our study is that increasing IP3 concentration increases calcium transient duration but may not always lead to positive inotropy. This work,together with the recent study by Hannanta-anan and Chow, suggests a plausible mechanism for IP3-dependent hypertrophic signalling in cardiomyocytes.
Viewpoint Optimization for Autonomous Strawberry Harvesting with Deep Reinforcement Learning

Autonomous harvesting may provide a viable solution to mounting labor pressures in the United States's strawberry industry. However, due to bottlenecks in machine perception and economic viability, a profitable and commercially adopted strawberry harvesting system remains elusive. In this research, we explore the feasibility of using deep reinforcement learning to overcome these bottlenecks and develop a practical algorithm to address the sub-objective of viewpoint optimization, or the development of a control policy to direct a camera to favorable vantage points for autonomous harvesting. We evaluate the algorithm's performance in a custom, open-source simulated environment and observe encouraging results. Our trained agent yields 8.7 times higher returns than random actions and 8.8 percent faster exploration than our best baseline policy, which uses visual servoing. Visual investigation shows the agent is able to fixate on favorable viewpoints, despite having no explicit means to propagate information through time. Overall, we conclude that deep reinforcement learning is a promising area of research to advance the state of the art in autonomous strawberry harvesting.
Evolutionary Dynamics of Cultural Memes and Application to Massive Movie Data

The profound impact of Darwin's theory of evolution on biology has led to the acceptance of the theory in many complex systems that lie well beyond its original domain. Culture is one example that also exhibits key Darwinian evolutionary properties: Differential adoption of cultural variants (variation and selection), new entities imitating older ones (inheritance), and convergence toward the most suitable state (adaptation). In this work we present a framework for capturing the details of the evolutionary dynamics in cultural systems on the "meme"- the cultural analog of the biological gene-level, and analyze large-scale, comprehensive movie-meme association data to construct a timeline of the history of cinema via the evolution of genres and the rise and fall of prominent sub-genres. We also identify the impactful movies that were harbingers to popular memes that we may say correspond to the proverbial "Eve" of the human race, shining light on the process by which certain genres form and grow. Finally, we measure how the impact of movies correlates with the experts' and the public's assessment.
LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models

We develop a new Low-level, First-order Probabilistic Programming Language~(LF-PPL) suited for models containing a mix of continuous, discrete, and/or piecewise-continuous variables. The key success of this language and its compilation scheme is in its ability to automatically distinguish parameters the density function is discontinuous with respect to, while further providing runtime checks for boundary crossings. This enables the introduction of new inference engines that are able to exploit gradient information, while remaining efficient for models which are not everywhere differentiable. We demonstrate this ability by incorporating a discontinuous Hamiltonian Monte Carlo (DHMC) inference engine that is able to deliver automated and efficient inference for non-differentiable models. Our system is backed up by a mathematical formalism that ensures that any model expressed in this language has a density with measure zero discontinuities to maintain the validity of the inference engine.
DEEPTURBO: Deep Turbo Decoder

Present-day communication systems routinely use codes that approach the channel capacity when coupled with a computationally efficient decoder. However, the decoder is typically designed for the Gaussian noise channel, and is known to be sub-optimal for non-Gaussian noise distribution. Deep learning methods offer a new approach for designing decoders that can be trained and tailored for arbitrary channel statistics. We focus on Turbo codes, and propose (DEEPTURBO), a novel deep learning based architecture for Turbo decoding. The standard Turbo decoder (TURBO) iteratively applies the Bahl-Cocke-Jelinek-Raviv (BCJR) algorithm with an interleaver in the middle. A neural architecture for TURBO decoding, termed (NEURALBCJR), was proposed recently to create a module that imitates the BCJR algorithm using supervised learning, and to use the interleaver architecture along with this module, which is then fine-tuned using end-to-end training. However, knowledge of the BCJR algorithm is required to design such an architecture, which also constrains the resulting learnt decoder. Here we remedy this requirement and propose a fully end-to-end trained neural decoder - Deep Turbo Decoder (DEEPTURBO). With novel learnable decoder structure and training methodology, DEEPTURBO reveals superior performance under both AWGN and non-AWGN settings as compared to the other two decoders - TURBOand NEURALBCJR.
SNU IDS at SemEval-2019 Task 3: Addressing Training-Test Class Distribution Mismatch in Conversational Classification

We present several techniques to tackle the mismatch in class distributions between training and test data in the Contextual Emotion Detection task of SemEval 2019, by extending the existing methods for class imbalance problem. Reducing the distance between the distribution of prediction and ground truth, they consistently show positive effects on the performance. Also we propose a novel neural architecture which utilizes representation of overall context as well as of each utterance. The combination of the methods and the models achieved micro F1 score of about 0.766 on the final evaluation.
Neural Empirical Bayes

We unify $\textit{kernel density estimation}$ and $\textit{empirical Bayes}$ and address a set of problems in unsupervised learning with a geometric interpretation of those methods, rooted in the $\textit{concentration of measure}$ phenomenon. Kernel density is viewed symbolically as $X\rightharpoonup Y$ where the random variable $X$ is smoothed to $Y= X+N(0,\sigma^2 I_d)$, and empirical Bayes is the machinery to denoise in a least-squares sense, which we express as $X \leftharpoondown Y$. A learning objective is derived by combining these two, symbolically captured by $X \rightleftharpoons Y$. Crucially, instead of using the original nonparametric estimators, we parametrize $\textit{the energy function}$ with a neural network denoted by $\phi$; at optimality, $\nabla \phi \approx -\nabla \log f$ where $f$ is the density of $Y$. The optimization problem is abstracted as interactions of high-dimensional spheres which emerge due to the concentration of isotropic gaussians. We introduce two algorithmic frameworks based on this machinery: (i) a "walk-jump" sampling scheme that combines Langevin MCMC (walks) and empirical Bayes (jumps), and (ii) a probabilistic framework for $\textit{associative memory}$, called NEBULA, defined a la Hopfield by the $\textit{gradient flow}$ of the learned energy to a set of attractors. We finish the paper by reporting the emergence of very rich "creative memories" as attractors of NEBULA for highly-overlapping spheres.
Compressing complex convolutional neural network based on an improved deep compression algorithm

Although convolutional neural network (CNN) has made great progress, large redundant parameters restrict its deployment on embedded devices, especially mobile devices. The recent compression works are focused on real-value convolutional neural network (Real CNN), however, to our knowledge, there is no attempt for the compression of complex-value convolutional neural network (Complex CNN). Compared with the real-valued network, the complex-value neural network is easier to optimize, generalize, and has better learning potential. This paper extends the commonly used deep compression algorithm from real domain to complex domain and proposes an improved deep compression algorithm for the compression of Complex CNN. The proposed algorithm compresses the network about 8 times on CIFAR-10 dataset with less than 3% accuracy loss. On the ImageNet dataset, our method compresses the model about 16 times and the accuracy loss is about 2% without retraining.
Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry

Training accurate 3D human pose estimators requires large amount of 3D ground-truth data which is costly to collect. Various weakly or self supervised pose estimation methods have been proposed due to lack of 3D data. Nevertheless, these methods, in addition to 2D ground-truth poses, require either additional supervision in various forms (e.g. unpaired 3D ground truth data, a small subset of labels) or the camera parameters in multiview settings. To address these problems, we present EpipolarPose, a self-supervised learning method for 3D human pose estimation, which does not need any 3D ground-truth data or camera extrinsics. During training, EpipolarPose estimates 2D poses from multi-view images, and then, utilizes epipolar geometry to obtain a 3D pose and camera geometry which are subsequently used to train a 3D pose estimator. We demonstrate the effectiveness of our approach on standard benchmark datasets (i.e. Human3.6M and MPI-INF-3DHP) where we set the new state-of-the-art among weakly/self-supervised methods. Furthermore, we propose a new performance measure Pose Structure Score (PSS) which is a scale invariant, structure aware measure to evaluate the structural plausibility of a pose with respect to its ground truth. Code and pretrained models are available at https://github.com/mkocabas/EpipolarPose