OpenRaiser/Intern-Atlas huggingface dataset:
DCVQE: A Hierarchical Transformer for Video Quality Assessment
Venue: arXiv cs.CV 2022
Year: 2022

The explosion of user-generated videos stimulates a great demand for no-reference video quality assessment (NR-VQA). Inspired by our observation on the actions of human annotation, we put forward a Divide and Conquer Video Quality Estimator (DCVQE) for NR-VQA. Starting from extracting the frame-level quality embeddings (QE), our proposal splits the whole sequence into a number of clips and applies Transformers to learn the clip-level QE and update the frame-level QE simultaneously; another Transformer is introduced to combine the clip-level QE to generate the video-level QE. We call this hierarchical combination of Transformers as a Divide and Conquer Transformer (DCTr) layer. An accurate video quality feature extraction can be achieved by repeating the process of this DCTr layer several times. Taking the order relationship among the annotated data into account, we also propose a novel correlation loss term for model training. Experiments on various datasets confirm the effectiveness and robustness of our DCVQE model.
Predicting Blossom Date of Cherry Tree With Support Vector Machine and Recurrent Neural Network
Venue: arXiv cs.LG 2022
Year: 2022

Our project probes the relationship between temperatures and the blossom date of cherry trees. Through modeling, future flowering will become predictive, helping the public plan travels and avoid pollen season. To predict the date when the cherry trees will blossom exactly could be viewed as a multiclass classification problem, so we applied the multi-class Support Vector Classifier (SVC) and Recurrent Neural Network (RNN), particularly Long Short-term Memory (LSTM), to formulate the problem. In the end, we evaluate and compare the performance of these approaches to find out which one might be more applicable in reality.
Improving Code Review with GitHub Issue Tracking
Venue: arXiv cs.SI 2022
Year: 2022

Software quality is an important problem for technology companies, since it substantially impacts the efficiency, usefulness, and maintainability of the final product; hence, code review is a must-do activity for software developers. During the code review process, senior engineers monitor other developers' work to spot possible problems and enforce coding standards. One of the most widely used open-source software platforms, GitHub, attracts millions of developers who use it to store their projects. This study aims to analyze code quality on GitHub from the standpoint of code reviews. We examined the code review process using GitHub's Issues Tracker, which allows team members to evaluate, discuss, and share their opinions on the proposed code before it is approved. Based on our analysis, we present a novel approach for improving the code review process by promoting regularity and community involvement.
HiCo: Hierarchical Contrastive Learning for Ultrasound Video Model Pretraining
Venue: arXiv cs.CV 2022
Year: 2022

The self-supervised ultrasound (US) video model pretraining can use a small amount of labeled data to achieve one of the most promising results on US diagnosis. However, it does not take full advantage of multi-level knowledge for learning deep neural networks (DNNs), and thus is difficult to learn transferable feature representations. This work proposes a hierarchical contrastive learning (HiCo) method to improve the transferability for the US video model pretraining. HiCo introduces both peer-level semantic alignment and cross-level semantic alignment to facilitate the interaction between different semantic levels, which can effectively accelerate the convergence speed, leading to better generalization and adaptation of the learned model. Additionally, a softened objective function is implemented by smoothing the hard labels, which can alleviate the negative effect caused by local similarities of images between different classes. Experiments with HiCo on five datasets demonstrate its favorable results over state-of-the-art approaches. The source code of this work is publicly available at https://github.com/983632847/HiCo.
Adaptive shape optimization with NURBS designs and PHT-splines for solution approximation in time-harmonic acoustics
Venue: arXiv cs.CE 2022
Year: 2022

Geometry Independent Field approximaTion (GIFT) was proposed as a generalization of Isogeometric analysis (IGA), where different types of splines are used for the parameterization of the computational domain and approximation of the unknown solution. GIFT with Non-Uniform Rational B-Splines (NUBRS) for the geometry and PHT-splines for the solution approximation were successfully applied to problems of time-harmonic acoustics, where it was shown that in some cases, adaptive PHT-spline mesh yields highly accurate solutions at lower computational cost than methods with uniform refinement. Therefore, it is of interest to investigate performance of GIFT for shape optimization problems, where NURBS are used to model the boundary with their control points being the design variables and PHT-splines are used to approximate the solution adaptively to the boundary changes during the optimization process. In this work we demonstrate the application of GIFT for 2D acoustic shape optimization problems and, using three benchmark examples, we show that the method yields accurate solutions with significant computational savings in terms of the number of degrees of freedom and computational time.
Bridging CLIP and StyleGAN through Latent Alignment for Image Editing
Venue: arXiv cs.CV 2022
Year: 2022

Text-driven image manipulation is developed since the vision-language model (CLIP) has been proposed. Previous work has adopted CLIP to design a text-image consistency-based objective to address this issue. However, these methods require either test-time optimization or image feature cluster analysis for single-mode manipulation direction. In this paper, we manage to achieve inference-time optimization-free diverse manipulation direction mining by bridging CLIP and StyleGAN through Latent Alignment (CSLA). More specifically, our efforts consist of three parts: 1) a data-free training strategy to train latent mappers to bridge the latent space of CLIP and StyleGAN; 2) for more precise mapping, temporal relative consistency is proposed to address the knowledge distribution bias problem among different latent spaces; 3) to refine the mapped latent in s space, adaptive style mixing is also proposed. With this mapping scheme, we can achieve GAN inversion, text-to-image generation and text-driven image manipulation. Qualitative and quantitative comparisons are made to demonstrate the effectiveness of our method.
Self-Supervised 3D Human Pose Estimation in Static Video Via Neural Rendering
Venue: arXiv cs.CV 2022
Year: 2022

Inferring 3D human pose from 2D images is a challenging and long-standing problem in the field of computer vision with many applications including motion capture, virtual reality, surveillance or gait analysis for sports and medicine. We present preliminary results for a method to estimate 3D pose from 2D video containing a single person and a static background without the need for any manual landmark annotations. We achieve this by formulating a simple yet effective self-supervision task: our model is required to reconstruct a random frame of a video given a frame from another timepoint and a rendered image of a transformed human shape template. Crucially for optimisation, our ray casting based rendering pipeline is fully differentiable, enabling end to end training solely based on the reconstruction task.
Everything is Varied: The Surprising Impact of Individual Variation on ML Robustness in Medicine
Venue: arXiv cs.LG 2022
Year: 2022

In medical settings, Individual Variation (IV) refers to variation that is due not to population differences or errors, but rather to within-subject variation, that is the intrinsic and characteristic patterns of variation pertaining to a given instance or the measurement process. While taking into account IV has been deemed critical for proper analysis of medical data, this source of uncertainty and its impact on robustness have so far been neglected in Machine Learning (ML). To fill this gap, we look at how IV affects ML performance and generalization and how its impact can be mitigated. Specifically, we provide a methodological contribution to formalize the problem of IV in the statistical learning framework and, through an experiment based on one of the largest real-world laboratory medicine datasets for the problem of COVID-19 diagnosis, we show that: 1) common state-of-the-art ML models are severely impacted by the presence of IV in data; and 2) advanced learning strategies, based on data augmentation and data imprecisiation, and proper study designs can be effective at improving robustness to IV. Our findings demonstrate the critical relevance of correctly accounting for IV to enable safe deployment of ML in clinical settings.
Using Detection, Tracking and Prediction in Visual SLAM to Achieve Real-time Semantic Mapping of Dynamic Scenarios
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we propose a lightweight system, RDS-SLAM, based on ORB-SLAM2, which can accurately estimate poses and build semantic maps at object level for dynamic scenarios in real time using only one commonly used Intel Core i7 CPU. In RDS-SLAM, three major improvements, as well as major architectural modifications, are proposed to overcome the limitations of ORB-SLAM2. Firstly, it adopts a lightweight object detection neural network in key frames. Secondly, an efficient tracking and prediction mechanism is embedded into the system to remove the feature points belonging to movable objects in all incoming frames. Thirdly, a semantic octree map is built by probabilistic fusion of detection and tracking results, which enables a robot to maintain a semantic description at object level for potential interactions in dynamic scenarios. We evaluate RDS-SLAM in TUM RGB-D dataset, and experimental results show that RDS-SLAM can run with 30.3 ms per frame in dynamic scenarios using only an Intel Core i7 CPU, and achieves comparable accuracy compared with the state-of-the-art SLAM systems which heavily rely on both Intel Core i7 CPUs and powerful GPUs.
Systematic Evaluation and User Study of Privacy of Default Apps in Apple's Mobile Ecosystem
Venue: arXiv cs.HC 2022
Year: 2022

Users need to configure default apps when they first start using their devices. The privacy configurations of the default apps do not always match what users think they have initially enabled. We first systematically evaluated the privacy configurations of default apps. We discovered serious issues with the documentation of the default apps. Based on these findings, we explored users' experiences with an interview study (N=15). Our findings from both studies show that: the instructions of setting privacy configurations of default apps are vague and lack required steps; users were unable to disable default apps from accessing their personal information; users assumed they were being tracked by some default apps; default apps may cause tensions in family relationships because of information sharing. Our results illuminate on the privacy and security implications of configuring the privacy of default apps and how users perceive and understand the mobile ecosystem.
The Rectilinear Steiner Forest Arborescence problem
Venue: arXiv cs.CG 2022
Year: 2022

Let $r$ be a point in the first quadrant $Q_1$ of the plane $\mathbb{R}^2$ and let $P \subset Q_1$ be a set of points such that for any $p \in P$, its $x$- and $y$-coordinate is at least as that of $r$. A rectilinear Steiner arborescence for $P$ with the root $r$ is a rectilinear Steiner tree $T$ for $P \cup \{r\}$ such that for each point $p \in P$, the length of the (unique) path in $T$ from $p$ to the root $r$ equals $({\rm x}(p)-{\rm x}(r))+({\rm y}(p))-({\rm y}(r))$, where ${\rm x}(q)$ and ${\rm y}(q)$ denote the $x$- and $y$-coordinate, respectively, of point $q \in P \cup \{r\}$. Given two point sets $P$ and $R$ lying in the first quadrant $Q_1$ and such that $(0,0) \in R$, the Rectilinear Steiner Forest Arborescence (RSFA) problem is to find the minimum-length spanning forest $F$ such that each connected component $F$ is a rectilinear Steiner arborescence rooted at some root in $R$. The RSFA problem is a natural generalization of the Rectilinear Steiner Arborescence problem, where $R=\{(0,0)\}$, and thus it is NP-hard. In this paper, we provide a simple exact exponential time algorithm for the RSFA problem, design a polynomial time approximation scheme as well as a fixed-parameter algorithm.
When Infodemic Meets Epidemic: a Systematic Literature Review
Venue: arXiv cs.SI 2022
Year: 2022

Epidemics and outbreaks present arduous challenges requiring both individual and communal efforts. Social media offer significant amounts of data that can be leveraged for bio-surveillance. They also provide a platform to quickly and efficiently reach a sizeable percentage of the population, hence their potential impact on various aspects of epidemic mitigation. The general objective of this systematic literature review is to provide a methodical overview of the integration of social media in different epidemic-related contexts. Three research questions were conceptualized for this review, resulting in over 10000 publications collected in the first PRISMA stage, 129 of which were selected for inclusion. A thematic method-oriented synthesis was undertaken and identified 5 main themes related to social media enabled epidemic surveillance, misinformation management, and mental health. Findings uncover a need for more robust applications of the lessons learned from epidemic post-mortem documentation. A vast gap exists between retrospective analysis of epidemic management and result integration in prospective studies. Harnessing the full potential of social media in epidemic related tasks requires streamlining the results of epidemic forecasting, public opinion understanding and misinformation propagation, all while keeping abreast of potential mental health implications. Pro-active prevention has thus become vital for epidemic curtailment and containment.
On Parallel or Distributed Asynchronous Iterations with Unbounded Delays and Possible Out of Order Messages or Flexible Communication for Convex Optimization Problems and Machine Learning
Venue: arXiv cs.DC 2022
Year: 2022

We describe several features of parallel or distributed asynchronous iterative algorithms such as unbounded delays, possible out of order messages or flexible communication. We concentrate on the concept of macroiteration sequence which was introduced in order to study the convergence or termination of asynchronous iterations. A survey of asynchronous iterations for convex optimization problems is also presented. Finally, a new result of convergence for parallel or distributed asynchronous iterative algorithms with flexible communication for convex optimization problems and machine learning is proposed.
A Prospective Analysis of Security Vulnerabilities within Link Traversal-Based Query Processing (Extended Version)
Venue: arXiv cs.CR 2022
Year: 2022

The societal and economical consequences surrounding Big Data-driven platforms have increased the call for decentralized solutions. However, retrieving and querying data in more decentralized environments requires fundamentally different approaches, whose properties are not yet well understood. Link Traversal-based Query Processing (LTQP) is a technique for querying over decentralized data networks, in which a client-side query engine discovers data by traversing links between documents. Since decentralized environments are potentially unsafe due to their non-centrally controlled nature, there is a need for client-side LTQP query engines to be resistant against security threats aimed at the query engine's host machine or the query initiator's personal data. As such, we have performed an analysis of potential security vulnerabilities of LTQP. This article provides an overview of security threats in related domains, which are used as inspiration for the identification of 10 LTQP security threats. Each threat is explained, together with an example, and one or more avenues for mitigations are proposed. We conclude with several concrete recommendations for LTQP query engine developers and data publishers as a first step to mitigate some of these issues. With this work, we start filling the unknowns for enabling querying over decentralized environments. Aside from future work on security, wider research is needed to uncover missing building blocks for enabling true decentralization.
End-to-End QoS for the Open Source Safety-Relevant RISC-V SELENE Platform
Venue: arXiv cs.AR 2022
Year: 2022

This paper presents the end-to-end QoS approach to provide performance guarantees followed in the SELENE platform, a high-performance RISC-V based heterogeneous SoC for safety-related real-time systems. Our QoS approach includes smart interconnect solutions for buses and NoCs, along with multicore interference-aware statistics units to, cooperatively, achieve end-to-end QoS.
Utilizing Explainable AI for improving the Performance of Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

Nowadays, deep neural networks are widely used in a variety of fields that have a direct impact on society. Although those models typically show outstanding performance, they have been used for a long time as black boxes. To address this, Explainable Artificial Intelligence (XAI) has been developing as a field that aims to improve the transparency of the model and increase their trustworthiness. We propose a retraining pipeline that consistently improves the model predictions starting from XAI and utilizing state-of-the-art techniques. To do that, we use the XAI results, namely SHapley Additive exPlanations (SHAP) values, to give specific training weights to the data samples. This leads to an improved training of the model and, consequently, better performance. In order to benchmark our method, we evaluate it on both real-life and public datasets. First, we perform the method on a radar-based people counting scenario. Afterward, we test it on the CIFAR-10, a public Computer Vision dataset. Experiments using the SHAP-based retraining approach achieve a 4% more accuracy w.r.t. the standard equal weight retraining for people counting tasks. Moreover, on the CIFAR-10, our SHAP-based weighting strategy ends up with a 3% accuracy rate than the training procedure with equal weighted samples.
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
Venue: arXiv cs.CL 2022
Year: 2022

Pre-trained language models (PLMs) have gained increasing popularity due to their compelling prediction performance in diverse natural language processing (NLP) tasks. When formulating a PLM-based prediction pipeline for NLP tasks, it is also crucial for the pipeline to minimize the calibration error, especially in safety-critical applications. That is, the pipeline should reliably indicate when we can trust its predictions. In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more. Although prior work has looked into some of these considerations, they usually draw conclusions based on a limited scope of empirical studies. There still lacks a holistic analysis on how to compose a well-calibrated PLM-based prediction pipeline. To fill this void, we compare a wide range of popular options for each consideration based on three prevalent NLP classification tasks and the setting of domain shift. In response, we recommend the following: (1) use ELECTRA for PLM encoding, (2) use larger PLMs if possible, (3) use Temp Scaling as the uncertainty quantifier, and (4) use Focal Loss for fine-tuning.
Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment
Venue: arXiv cs.CV 2022
Year: 2022

Multimedia summarization with multimodal output (MSMO) is a recently explored application in language grounding. It plays an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. However, existing methods extract features from the whole video and article and use fusion methods to select the representative one, thus usually ignoring the critical structure and varying semantics. In this work, we propose a Semantics-Consistent Cross-domain Summarization (SCCS) model based on optimal transport alignment with visual and textual segmentation. In specific, our method first decomposes both video and article into segments in order to capture the structural semantics, respectively. Then SCCS follows a cross-domain alignment objective with optimal transport distance, which leverages multimodal interaction to match and select the visual and textual summary. We evaluated our method on three recent multimodal datasets and demonstrated the effectiveness of our method in producing high-quality multimodal summaries.
Tango or Square Dance? How Tightly Should we Integrate Network Functionality in Browsers?
Venue: arXiv cs.NI 2022
Year: 2022

The question at which layer network functionality is presented or abstracted remains a research challenge. Traditionally, network functionality was either placed into the core network, middleboxes, or into the operating system -- but recent developments have expanded the design space to directly introduce functionality into the application (and in particular into the browser) as a way to expose it to the user. Given the context of emerging path-aware networking technology, an interesting question arises: which layer should handle the new features? We argue that the browser is becoming a powerful platform for network innovation, where even user-driven properties can be implemented in an OS-agnostic fashion. We demonstrate the feasibility of geo-fenced browsing using a prototype browser extension, realized by the SCION path-aware networking architecture, without introducing any significant performance overheads.
TensorFlow as a DSL for stencil-based computation on the Cerebras Wafer Scale Engine
Venue: arXiv cs.DC 2022
Year: 2022

The Cerebras Wafer Scale Engine (WSE) is an accelerator that combines hundreds of thousands of AI-cores onto a single chip. Whilst this technology has been designed for machine learning workloads, the significant amount of available raw compute means that it is also a very interesting potential target for accelerating traditional HPC computational codes. Many of these algorithms are stencil-based, where update operations involve contributions from neighbouring elements, and in this paper we explore the suitability of this technology for such codes from the perspective of an early adopter of the technology, compared to CPUs and GPUs. Using TensorFlow as the interface, we explore the performance and demonstrate that, whilst there is still work to be done around exposing the programming interface to users, performance of the WSE is impressive as it out performs four V100 GPUs by two and a half times and two Intel Xeon Platinum CPUs by around 114 times in our experiments. There is significant potential therefore for this technology to play an important role in accelerating HPC codes on future exascale supercomputers.