OpenRaiser/Intern-Atlas huggingface dataset:
"When Words Fail, Emojis Prevail": Generating Sarcastic Utterances with Emoji Using Valence Reversal and Semantic Incongruity
Venue: arXiv cs.CL 2023
Year: 2023

Sarcasm is a form of figurative language that serves as a humorous tool for mockery and ridicule. We present a novel architecture for sarcasm generation with emoji from a non-sarcastic input sentence in English. We divide the generation task into two sub tasks: one for generating textual sarcasm and another for collecting emojis associated with those sarcastic sentences. Two key elements of sarcasm are incorporated into the textual sarcasm generation task: valence reversal and semantic incongruity with context, where the context may involve shared commonsense or general knowledge between the speaker and their audience. The majority of existing sarcasm generation works have focused on this textual form. However, in the real world, when written texts fall short of effectively capturing the emotional cues of spoken and face-to-face communication, people often opt for emojis to accurately express their emotions. Due to the wide range of applications of emojis, incorporating appropriate emojis to generate textual sarcastic sentences helps advance sarcasm generation. We conclude our study by evaluating the generated sarcastic sentences using human judgement. All the codes and data used in this study has been made publicly available.
Whats next? Forecasting scientific research trends
Venue: arXiv cs.DL 2023
Year: 2023

Scientific research trends and interests evolve over time. The ability to identify and forecast these trends is vital for educational institutions, practitioners, investors, and funding organizations. In this study, we predict future trends in scientific publications using heterogeneous sources, including historical publication time series from PubMed, research and review articles, pre-trained language models, and patents. We demonstrate that scientific topic popularity levels and changes (trends) can be predicted five years in advance across 40 years and 125 diverse topics, including life-science concepts, biomedical, anatomy, and other science, technology, and engineering topics. Preceding publications and future patents are leading indicators for emerging scientific topics. We find the ratio of reviews to original research articles informative for identifying increasing or declining topics, with declining topics having an excess of reviews. We find that language models provide improved insights and predictions into temporal dynamics. In temporal validation, our models substantially outperform the historical baseline. Our findings suggest that similar dynamics apply across other scientific and engineering research topics. We present SciTrends, a user-friendly webtool for predicting scientific topics trends: https://hadasakaufman.shinyapps.io/SciTrend
Maintaining Stability and Plasticity for Predictive Churn Reduction
Venue: arXiv cs.LG 2023
Year: 2023

Deployed machine learning models should be updated to take advantage of a larger sample size to improve performance, as more data is gathered over time. Unfortunately, even when model updates improve aggregate metrics such as accuracy, they can lead to errors on samples that were correctly predicted by the previous model causing per-sample regression in performance known as predictive churn. Such prediction flips erode user trust thereby reducing the effectiveness of the human-AI team as a whole. We propose a solution called Accumulated Model Combination (AMC) based keeping the previous and current model version, and generating a meta-output using the prediction of the two models. AMC is a general technique and we propose several instances of it, each having their own advantages depending on the model and data properties. AMC requires minimal additional computation and changes to training procedures. We motivate the need for AMC by showing the difficulty of making a single model consistent with its own predictions throughout training thereby revealing an implicit stability-plasticity tradeoff when training a single model. We demonstrate the effectiveness of AMC on a variety of modalities including computer vision, text, and tabular datasets comparing against state-of-the-art churn reduction methods, and showing superior churn reduction ability compared to all existing methods while being more efficient than ensembles.
On the usefulness of linear types for correct nonce use enforcement during compile time
Venue: arXiv cs.CR 2023
Year: 2023

Cryptographic algorithms and protocols often need unique random numbers as parameters (e.g. nonces). Failure to satisfy this requirement lead to vulnerable implementation and can result in security breach. We show how linear types and static type checking can be used to enforce the correct generation of a new unique random number for each function invocation.
A Novel Reward Shaping Function for Single-Player Mahjong
Venue: arXiv cs.GT 2023
Year: 2023

Mahjong is a complex game with an intractably large state space with extremely sparse rewards, which poses challenges to develop an agent to play Mahjong. To overcome this, the ShangTing function was adopted as a reward shaping function. This was combined with a forward-search algorithm to create an agent capable of completing a winning hand in Single-player Mahjong (an average of 35 actions over 10,000 games). To increase performance, we propose a novel bonus reward shaping function, which assigns higher relative values to synergistic Mahjong hands. In a simulated 1-v-1 battle, usage of the new reward function outperformed the default ShangTing function, winning an average of $1.37 over 1000 games.
Video-Specific Query-Key Attention Modeling for Weakly-Supervised Temporal Action Localization
Venue: arXiv cs.CV 2023
Year: 2023

Weakly-supervised temporal action localization aims to identify and localize the action instances in the untrimmed videos with only video-level action labels. When humans watch videos, we can adapt our abstract-level knowledge about actions in different video scenarios and detect whether some actions are occurring. In this paper, we mimic how humans do and bring a new perspective for locating and identifying multiple actions in a video. We propose a network named VQK-Net with a video-specific query-key attention modeling that learns a unique query for each action category of each input video. The learned queries not only contain the actions' knowledge features at the abstract level but also have the ability to fit this knowledge into the target video scenario, and they will be used to detect the presence of the corresponding action along the temporal dimension. To better learn these action category queries, we exploit not only the features of the current input video but also the correlation between different videos through a novel video-specific action category query learner worked with a query similarity loss. Finally, we conduct extensive experiments on three commonly used datasets (THUMOS14, ActivityNet1.2, and ActivityNet1.3) and achieve state-of-the-art performance.
Using LOR Syringe Probes as a Method to Reduce Errors in Epidural Analgesia -- a Robotic Simulation Study
Venue: arXiv cs.RO 2023
Year: 2023

Epidural analgesia involves injection of anesthetics into the epidural space, using a Touhy needle to proceed through the layers in the epidural region and a "loss of resistance" (LOR) syringe to sense the environment stiffness. The anesthesiologist's case experience is one of the leading causes of accidental dural puncture and failed epidural - the two most common complications of epidural analgesia. Robotic simulation is an appealing solution to help train novices in this task. Another benefit of it is the ability to record the kinematic information throughout the procedure. In this work, we used a haptic bimanual simulator, that we designed and validated in previous work, to explore the effect LOR probing strategies had on procedure outcomes. Our results indicate that most participants probed more with the LOR syringe in successful trials, compared to unsuccessful trials. Furthermore, this result was more prominent in the three layers preceding the epidural space. Our findings can assist in creating better instructions for training novices in the task of epidural analgesia. We posit that instructing anesthesia residents to use the LOR syringe more extensively and educating them to do so more when they are in proximity to the epidural space can help improve skill acquisition in this task.
MrTF: Model Refinery for Transductive Federated Learning
Venue: arXiv cs.LG 2023
Year: 2023

We consider a real-world scenario in which a newly-established pilot project needs to make inferences for newly-collected data with the help of other parties under privacy protection policies. Current federated learning (FL) paradigms are devoted to solving the data heterogeneity problem without considering the to-be-inferred data. We propose a novel learning paradigm named transductive federated learning (TFL) to simultaneously consider the structural information of the to-be-inferred data. On the one hand, the server could use the pre-available test samples to refine the aggregated models for robust model fusion, which tackles the data heterogeneity problem in FL. On the other hand, the refinery process incorporates test samples into training and could generate better predictions in a transductive manner. We propose several techniques including stabilized teachers, rectified distillation, and clustered label refinery to facilitate the model refinery process. Abundant experimental studies verify the superiorities of the proposed \underline{M}odel \underline{r}efinery framework for \underline{T}ransductive \underline{F}ederated learning (MrTF). The source code is available at \url{https://github.com/lxcnju/MrTF}.
Bi-Mapper: Holistic BEV Semantic Mapping for Autonomous Driving
Venue: arXiv cs.CV 2023
Year: 2023

A semantic map of the road scene, covering fundamental road elements, is an essential ingredient in autonomous driving systems. It provides important perception foundations for positioning and planning when rendered in the Bird's-Eye-View (BEV). Currently, the prior knowledge of hypothetical depth can guide the learning of translating front perspective views into BEV directly with the help of calibration parameters. However, it suffers from geometric distortions in the representation of distant objects. In addition, another stream of methods without prior knowledge can learn the transformation between front perspective views and BEV implicitly with a global view. Considering that the fusion of different learning methods may bring surprising beneficial effects, we propose a Bi-Mapper framework for top-down road-scene semantic understanding, which incorporates a global view and local prior knowledge. To enhance reliable interaction between them, an asynchronous mutual learning strategy is proposed. At the same time, an Across-Space Loss (ASL) is designed to mitigate the negative impact of geometric distortions. Extensive results on nuScenes and Cam2BEV datasets verify the consistent effectiveness of each module in the proposed Bi-Mapper framework. Compared with exiting road mapping networks, the proposed Bi-Mapper achieves 2.1% higher IoU on the nuScenes dataset. Moreover, we verify the generalization performance of Bi-Mapper in a real-world driving scenario. The source code is publicly available at https://github.com/lynn-yu/Bi-Mapper.
RFR-WWANet: Weighted Window Attention-Based Recovery Feature Resolution Network for Unsupervised Image Registration
Venue: arXiv cs.CV 2023
Year: 2023

The Swin transformer has recently attracted attention in medical image analysis due to its computational efficiency and long-range modeling capability. Owing to these properties, the Swin Transformer is suitable for establishing more distant relationships between corresponding voxels in different positions in complex abdominal image registration tasks. However, the registration models based on transformers combine multiple voxels into a single semantic token. This merging process limits the transformers to model and generate coarse-grained spatial information. To address this issue, we propose Recovery Feature Resolution Network (RFRNet), which allows the transformer to contribute fine-grained spatial information and rich semantic correspondences to higher resolution levels. Furthermore, shifted window partitioning operations are inflexible, indicating that they cannot perceive the semantic information over uncertain distances and automatically bridge the global connections between windows. Therefore, we present a Weighted Window Attention (WWA) to build global interactions between windows automatically. It is implemented after the regular and cyclic shift window partitioning operations within the Swin transformer block. The proposed unsupervised deformable image registration model, named RFR-WWANet, detects the long-range correlations, and facilitates meaningful semantic relevance of anatomical structures. Qualitative and quantitative results show that RFR-WWANet achieves significant improvements over the current state-of-the-art methods. Ablation experiments demonstrate the effectiveness of the RFRNet and WWA designs. Our code is available at \url{https://github.com/MingR-Ma/RFR-WWANet}.
Estimation of control area in badminton doubles with pose information from top and back view drone videos
Venue: arXiv cs.CV 2023
Year: 2023

The application of visual tracking to the performance analysis of sports players in dynamic competitions is vital for effective coaching. In doubles matches, coordinated positioning is crucial for maintaining control of the court and minimizing opponents' scoring opportunities. The analysis of such teamwork plays a vital role in understanding the dynamics of the game. However, previous studies have primarily focused on analyzing and assessing singles players without considering occlusion in broadcast videos. These studies have relied on discrete representations, which involve the analysis and representation of specific actions (e.g., strokes) or events that occur during the game while overlooking the meaningful spatial distribution. In this work, we present the first annotated drone dataset from top and back views in badminton doubles and propose a framework to estimate the control area probability map, which can be used to evaluate teamwork performance. We present an efficient framework of deep neural networks that enables the calculation of full probability surfaces. This framework utilizes the embedding of a Gaussian mixture map of players' positions and employs graph convolution on their poses. In the experiment, we verify our approach by comparing various baselines and discovering the correlations between the score and control area. Additionally, we propose a practical application for assessing optimal positioning to provide instructions during a game. Our approach offers both visual and quantitative evaluations of players' movements, thereby providing valuable insights into doubles teamwork. The dataset and related project code is available at https://github.com/Ning-D/Drone_BD_ControlArea
From Unstructured to Structured: Transforming Chatbot Dialogues into Data Mart Schema for Visualization
Venue: arXiv cs.DB 2023
Year: 2023

Schools are among the primary avenues for public healthcare interventions. With resource limitations posing challenges to the routine conduct of health and wellness checks in Philippine public schools, the deployment of a chatbot-assisted health monitoring system may provide an alternative method. However, deriving insights from raw conversations is not straightforward due to the expressiveness of natural language that causes variances in the input. In this paper, we present a process for transforming unstructured dialogues into a structured schema. The process comprises four stages: (i) processing the dialogues through entity extraction and data aggregation, (ii) storing them as NoSQL documents on the cloud, (iii) transforming them into a star schema for online analytical processing and building an extract-transform-load workflow, and (iv) creating a web-based dashboard for visualizing summarized data and reports. Performance evaluation of this dashboard showed that increasing the number of stored dialogues by a factor of 100,000 increased the loading time for the display of roll-up, drill-down, and filter results by around only one second.
Poses as Queries: Image-to-LiDAR Map Localization with Transformers
Venue: arXiv cs.RO 2023
Year: 2023

High-precision vehicle localization with commercial setups is a crucial technique for high-level autonomous driving tasks. Localization with a monocular camera in LiDAR map is a newly emerged approach that achieves promising balance between cost and accuracy, but estimating pose by finding correspondences between such cross-modal sensor data is challenging, thereby damaging the localization accuracy. In this paper, we address the problem by proposing a novel Transformer-based neural network to register 2D images into 3D LiDAR map in an end-to-end manner. Poses are implicitly represented as high-dimensional feature vectors called pose queries and can be iteratively updated by interacting with the retrieved relevant information from cross-model features using attention mechanism in a proposed POse Estimator Transformer (POET) module. Moreover, we apply a multiple hypotheses aggregation method that estimates the final poses by performing parallel optimization on multiple randomly initialized pose queries to reduce the network uncertainty. Comprehensive analysis and experimental results on public benchmark conclude that the proposed image-to-LiDAR map localization network could achieve state-of-the-art performances in challenging cross-modal localization tasks.
A generalized network level disruption strategy selection model for urban public transport systems
Venue: arXiv cs.CY 2023
Year: 2023

A fast recovery from disruptions is of vital importance for the reliability of transit systems. This study presents a new attempt to tackle the transit disruption mitigation problem in a comprehensive and hierarchical way. A network level strategy selection optimization model is formulated as a joint routing and resource allocation (nJRRA) problem. By constraining the problem further into an epsilon-constrained nJRRA problem, classic solution algorithms can be applied to solve the quadratically constrained quadratic program (QCQP). On top of this "basic model", we propose adding a decision to delay the resource allocation decisions up to a maximum initiation time when the incident duration is stochastic. To test the models, a quasi-dynamic evaluation program with a given incident duration distribution is constructed using discretized time steps and discrete distributions. Five different demand patterns and four different disruption duration distributions (20 combinations) are tested on a toy transit network. The results show that the two models outperform benchmark strategies such as using only line level adjustment or only bus bridging. They also highlight conditions when delaying the decision is preferred.
Living in a Material World: Learning Material Properties from Full-Waveform Flash Lidar Data for Semantic Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

Advances in lidar technology have made the collection of 3D point clouds fast and easy. While most lidar sensors return per-point intensity (or reflectance) values along with range measurements, flash lidar sensors are able to provide information about the shape of the return pulse. The shape of the return waveform is affected by many factors, including the distance that the light pulse travels and the angle of incidence with a surface. Importantly, the shape of the return waveform also depends on the material properties of the reflecting surface. In this paper, we investigate whether the material type or class can be determined from the full-waveform response. First, as a proof of concept, we demonstrate that the extra information about material class, if known accurately, can improve performance on scene understanding tasks such as semantic segmentation. Next, we learn two different full-waveform material classifiers: a random forest classifier and a temporal convolutional neural network (TCN) classifier. We find that, in some cases, material types can be distinguished, and that the TCN generally performs better across a wider range of materials. However, factors such as angle of incidence, material colour, and material similarity may hinder overall performance.
Towards a More Inclusive Metaverse via Designing Tools That Support Collaborative Virtual World Building by Users With and Without Disabilities
Venue: arXiv cs.HC 2023
Year: 2023

Research has found social VR to bring various benefits to users with and without disabilities. Given the success of social VR applications that support user-created worlds, it is important to consider how we can empower users in building inclusive virtual worlds by investigating how tools for world building can be built to better support collaborations between users with and without disabilities. As such, this position paper provides a brief discussion of existing research into important factors that should be considered during such collaborations, and possible future research directions.
Spatiotemporally Consistent HDR Indoor Lighting Estimation
Venue: arXiv cs.CV 2023
Year: 2023

We propose a physically-motivated deep learning framework to solve a general version of the challenging indoor lighting estimation problem. Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position. Particularly, when the input is an LDR video sequence, our framework not only progressively refines the lighting prediction as it sees more regions, but also preserves temporal consistency by keeping the refinement smooth. Our framework reconstructs a spherical Gaussian lighting volume (SGLV) through a tailored 3D encoder-decoder, which enables spatially consistent lighting prediction through volume ray tracing, a hybrid blending network for detailed environment maps, an in-network Monte-Carlo rendering layer to enhance photorealism for virtual object insertion, and recurrent neural networks (RNN) to achieve temporally consistent lighting prediction with a video sequence as the input. For training, we significantly enhance the OpenRooms public dataset of photorealistic synthetic indoor scenes with around 360K HDR environment maps of much higher resolution and 38K video sequences, rendered with GPU-based path tracing. Experiments show that our framework achieves lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods, leading to photorealistic AR applications such as object insertion.
A New Upper Bound on the Maximal Error Resilience of Interactive Error-Correcting Codes
Venue: arXiv cs.DS 2023
Year: 2023

In an interactive error-correcting code (iECC), Alice and Bob engage in an interactive protocol with the goal of Alice communicating a message $x \in \{ 0, 1 \}^k$ to Bob in such a way that even if some fraction of the total communicated bits are corrupted, Bob can still determine $x$. It was shown in works by Gupta, Kalai, and Zhang (STOC 2022) and by Efremenko, Kol, Saxena, and Zhang (FOCS 2022) that there exist iECCs that are resilient to a larger fraction of errors than is possible in standard error-correcting codes without interaction. One major question in the study of iECCs is to determine the optimal error resilience achievable by an iECC. In the case of bit flip errors, it is known that an iECC can achieve $\frac14 + 10^{-5}$ error resilience (Efremenko, Kol, Saxena, and Zhang), while the best known upper bound is $\frac27 \approx 0.2857$ (Gupta, Kalai, and Zhang). In this work, we improve upon the upper bound, showing that no iECC can be resilient to more than $\frac{13}{47} \approx 0.2766$ fraction of errors.