Miracle-dz/UniFAR-FTUs huggingface dataset:
How does Color Constancy Affect Target Recognition and Instance Segmentation?

Previous work has demonstrated that incorrect white balance (WB) in the camera image signal processing pipeline has a negative impact on the performance of deep neural networks (DNNs) in high-level vision tasks, and traditional image augmentation approaches are not well suited for modeling WB errors. However, it is still unclear when this impact will occur for which kinds of images and objects. In this paper, we manually labeled 2304 images from the RECommended dataset and NUS dataset and discovered that the effect of WB on DNNs is greatly associated with object size and occlusion level among objects. In images with incorrect WB, small objects and objects with heavily occluded backgrounds are the main factors resulting in the bad performance of DNNs, indicating that the effect of WB is clearly associated with the shape of objects. Our findings may support that the functional role of some neurons in the visual cortex (e.g., V1 or V4 areas) realizing color constancy (CC) and encoding object attributes such as color and shape dependently is to contribute to high-level vision. Furthermore, based on this scientific finding, we proposed a novel augmentation strategy to address the negative impact of incorrect WB by expanding the training datasets in both color transformation and synthetic occlusion. We compared our proposed strategy with the current augmentation strategies and showed that our approach clearly improves the performance of DNNs in detection and segmentation tasks with small objects and objects with heavily occluded backgrounds.
ArtiVisual: A Platform to Generate and Compare Art

ArtiVisual is a platform for generating new art-pieces based on an existing art style and comparing commonalities between paintings from different era. We combine an image generative network with established state-of-the-art visualisation techniques to deepen the users' understanding of art in general. With ArtiVisual we can generate images based on art- styles via an interactive timeline. Common features between art-styles are reflected on the generated art piece produced by the network after learning the subspace of each artist's specific features. Visualisations are presented to provide insight into commonalities between existing and generated images. The combination of a trained network and our visualisation techniques provides a rigid framework for thorough exploration and understanding of art datasets.
ITSLF: Inter-Thread Store-to-Load Forwardingin Simultaneous Multithreading

In this paper, we argue that, for a class of fine-grain, synchronization-intensive, parallel workloads, it is advantageous to consolidate synchronization and communication as much as possible among the threads of simultaneous multithreading (SMT) cores. While, today, the shared L1 is the closest coherent level where synchronization and communication between SMT threads can take place, we observe that there is an even closer shared level, entirely inside a single core. This level comprises the load queues (LQ) and store queues (SQ) / store buffers (SB) of the SMT threads and to the best of our knowledge it has never been used as such. The reason is that if we allow communication of different SMT threads via their LQs and SQs/SBs, i.e., inter-thread store-to-load forwarding (ITSLF), we violate write atomicity with respect to the outside world, beyond the acceptable model of read-own-write-early multiple-copy atomicity (rMCA). The key insight of our work is that we can accelerate synchronization and communication among SMT threads with inter-thread store-to-load forwarding, without affecting the memory model—in particular without violating rMCA. We demonstrate how we can achieve this entirely through speculative interactions between LQs and SQs/SBs of different threads, while ensuring deadlock-free execution. Without changing the architectural model, the ISA, or the software, and without adding extra hardware in the form of a specialized accelerator, our insight enables a new design point for a standard architecture. We demonstrate that with ITSLF, workloads scale better on a single 8-way SMT core (with the resources of a single-threaded core) than on a baseline SMT (with or without optimizations), or on 8 single-threaded cores.
NMAP: Power Management Based on Network Packet Processing Mode Transition for Latency-Critical Workloads

Processor power management exploiting Dynamic Voltage and Frequency Scaling (DVFS) plays a crucial role in improving the data-center’s energy efficiency. However, we observe that current power management policies in Linux (i.e., governors) often considerably increase tail response time (i.e., violate a given Service Level Objective (SLO)) and energy consumption of latency-critical applications. Furthermore, the previously proposed SLO-aware power management policies oversimplify network request processing and ignore the fact that network requests arrive at the application layer in bursts. Considering the complex interplay between the OS and network devices, we propose a power management framework exploiting network packet processing mode transitions in the OS to quickly react to the processing demands from the received network requests. Our proposed power management framework tracks the transitions between polling and interrupt in the network software stack to detect excessive packet processing on the cores and immediately react to the load changes by updating the voltage and frequency (V/F) states. Our experimental results show that our framework does not violate SLO and reduces energy consumption by up to 35.7% and 14.8% compared to Linux governors and state-of-the-art SLO-aware power management techniques, respectively.
Simplifying Multimodal Emotion Recognition with Single Eye Movement Modality

Multimodal emotion recognition has long been a popular topic in affective computing since it significantly enhances the performance compared with that of a single modality. Among all, the combination of electroencephalography (EEG) and eye movement signals is one of the most attractive practices due to their complementarity and objectivity. However, the high cost and inconvenience of EEG signal acquisition severely hamper the popularization of multimodal emotion recognition in practical scenarios, while eye movement signals are much easier to acquire. To increase the feasibility and the generalization ability of emotion decoding without compromising the performance, we propose a generative adversarial network-based framework. In our model, a single modality of eye movements is used as input and it is capable of mapping the information onto multimodal features. Experimental results on SEED series datasets with different emotion categories demonstrate that our model with multimodal features generated by the single eye movement modality maintains competitive accuracies compared to those with multimodality input and drastically outperforms those single-modal emotion classifiers. This illustrates that the model has the potential to reduce the dependence on multimodalities without sacrificing performance which makes emotion recognition more applicable and practicable.
Text to Scene: A System of Configurable 3D Indoor Scene Synthesis

In this work, we show the Text to Scene system, which can configure 3D indoor scene from natural language. Given a text, the system will organize inclusive semantic message to a graph template, complete the graph with a novel graph-based contextual completion method Contextual ConvE(CConvE) and visulize the graph by arranging 3D models under an object location protocol. In the experiments, qualitative results obtained by the Text to Scene(T2S) system and quantitative evaluation of CConvE compared with other state-of-the-art approaches are reported.
FTAFace: Context-enhanced Face Detector with Fine-grained Task Attention

In face detection, it is a common strategy to treat samples differently according to their difficulty for balancing training data distribution. However, we observe that widely used sampling strategies, such as OHEM and Focal loss, can lead to the performance imbalance between different tasks (e.g., classification and localization). Through analysis, we point out that, due to the driving of classification information, these sample-based strategies are difficult to coordinate the attention of different tasks during the training, thus leading to the above imbalance. Accordingly, we first confirm this by shifting the attention from the sample level to the task level. Then, we propose a fine-grained task attention method, a.k. a FTA, including inter-task importance and intra-task importance, which adaptively adjusts the attention of each item in the task from both global and local perspectives, so as to achieve finer optimization. In addition, we introduce transformer as a feature enhancer to assist our convolution network, and propose a context enhancement transformer, a.k. a CET, to mine the spatial relationship in the features towards more robust feature representation. Extensive experiments on WiderFace and FDDB benchmarks demonstrate that our method significantly boosts the baseline performance by 2.7%, 2.3% and 4.9% on easy, medium and hard validation sets respectively. Furthermore, the proposed FTAFace-light achieves higher accuracy than the state-of-the-art and reduces the amount of computation by 28.9%.
A Deeper Look into RowHammer’s Sensitivities: Experimental Analysis of Real DRAM Chips and Implications on Future Attacks and Defenses

RowHammer is a circuit-level DRAM vulnerability where repeatedly accessing (i.e., hammering) a DRAM row can cause bit flips in physically nearby rows. The RowHammer vulnerability worsens as DRAM cell size and cell-to-cell spacing shrink. Recent studies demonstrate that modern DRAM chips, including chips previously marketed as RowHammer-safe, are even more vulnerable to RowHammer than older chips such that the required hammer count to cause a bit flip has reduced by more than 10X in the last decade. Therefore, it is essential to develop a better understanding and in-depth insights into the RowHammer vulnerability of modern DRAM chips to more effectively secure current and future systems. Our goal in this paper is to provide insights into fundamental properties of the RowHammer vulnerability that are not yet rigorously studied by prior works, but can potentially be i) exploited to develop more effective RowHammer attacks or ii) leveraged to design more effective and efficient defense mechanisms. To this end, we present an experimental characterization using 248 DDR4 and 24 DDR3 modern DRAM chips from four major DRAM manufacturers demonstrating how the RowHammer effects vary with three fundamental properties: 1) DRAM chip temperature, 2) aggressor row active time, and 3) victim DRAM cell’s physical location. Among our 16 new observations, we highlight that a RowHammer bit flip 1) is very likely to occur in a bounded range, specific to each DRAM cell (e.g., 5.4% of the vulnerable DRAM cells exhibit errors in the range to ), 2) is more likely to occur if the aggressor row is active for longer time (e.g., RowHammer vulnerability increases by 36% if we keep a DRAM row active for 15 column accesses), and 3) is more likely to occur in certain physical regions of the DRAM module under attack (e.g., 5% of the rows are 2x more vulnerable than the remaining 95% of the rows). Our study has important practical implications on future RowHammer attacks and defenses. We describe and analyze the implications of our new findings by proposing three future RowHammer attack and five future RowHammer defense improvements.
Show, Read and Reason: Table Structure Recognition with Flexible Context Aggregator

We investigate the challenging problem of table structure recognition in this work. Many recent methods adopt graph-based context aggregator with strong inductive bias to reason sparse contextual relationships of table elements. However, the strong constraints may be too restrictive to represent the complicated table relationships. In order to learn more appropriate inductive bias from data, we try to introduce Transformer as context aggregator in this work. Nevertheless, Transformer taking dense context as input requires larger scale data and may suffer from unstable training procedure due to the weakening of inductive bias. To overcome the above limitations, we in this paper design a FLAG (FLexible context AGgregator), which marries Transformer with graph-based context aggregator in an adaptive way. Based on FLAG, an end-to-end framework requiring no extra meta-data or OCR information, termed FLAG-Net, is proposed to flexibly modulate the aggregation of dense context and sparse one for the relational reasoning of table elements. We investigate the modulation pattern in FLAG and show what contextual information is focused, which is vital for recognizing table structure. Extensive experimental results on benchmarks demonstrate the performance of our proposed FLAG-Net surpasses other compared methods by a large margin.
CaFGraph: Context-aware Facial Multi-graph Representation for Facial Action Unit Recognition

Facial action unit (AU) recognition has attracted increasing attention due to its indispensable role in affective computing, especially in the field of affective human-computer interaction. Due to the subtle and transient nature of AU, it is challenging to capture the delicate and ambiguous motions in local facial regions among consecutive frames. Considering that context is essential to resolve ambiguity in human visual system, modeling context within or among facial images emerges as a promising approach for AU recognition task. To this end, we propose CaFGraph, a novel context-aware facial multi-graph that can model both morphological & muscular-based region-level local context and region-level temporal context. CaFGraph is the first work to construct a universal facial multi-graph structure that is independent of both task settings and dataset statistics for almost all fine-grained facial behavior analysis tasks, including but not limited to AU recognition. To make full use of the context, we then present CaFNet that learns context-aware facial graph representations via CaFGraph from facial images for multi-label AU recognition. Experiments on two widely used benchmark datasets, BP4D and DISFA, demonstrate the superiority of our CaFNet over the state-of-the-art methods.
Regulating Personal Cameras for Disabled People and People with Deafblindness: Implications for HCI and Accessible Computing

In this experience paper we consider the relevance of social policy to the design of personal cameras for accessibility. As researchers with different backgrounds, in disability studies, accessible computing, AI-based systems, HCI and disability policy, we reflect broadly on our experiences of developing assistive technology for and with people with deafblindness. For designers of assistive technology there are usually few restrictions placed on what may be investigated, provided certain ethical and legal standards are met. However, deafblind and disabled people experience many more barriers in how the products of design may be accessed and how they may be used. Social policy is one of the mediators that governs the allocation of resources and benefits, especially for disabled people. We discuss these issues for researchers in the field, using the example of personal cameras: an area of high policy intervention. Awareness of policy is limited in HCI research, and we argue that it has the potential to add focus to work on design and assistive devices for disabled people. Designers have an important role to play in this process.
Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

Improving robustness against data missing has become one of the core challenges in Multimodal Sentiment Analysis (MSA), which aims to judge speaker sentiments from the language, visual, and acoustic signals. In the current research, translation-based methods and tensor regularization methods are proposed for MSA with incomplete modality features. However, both of them fail to cope with random modality feature missing in non-aligned sequences. In this paper, a transformer-based feature reconstruction network (TFR-Net) is proposed to improve the robustness of models for the random missing in non-aligned modality sequences. First, intra-modal and inter-modal attention-based extractors are adopted to learn robust representations for each element in modality sequences. Then, a reconstruction module is proposed to generate the missing modality features. With the supervision of SmoothL1Loss between generated and complete sequences, TFR-Net is expected to learn semantic-level features corresponding to missing features. Extensive experiments on two public benchmark datasets show that our model achieves good results against data missing across various missing modality combinations and various missing degrees.
Reconstruction: A Motion Driven Interactive Artwork Inspired by Chinese Shadow Puppet

Shadow puppet play is a representative Chinese intangible cultural heritage, which has a history of more than two thousand years. However, with the popularity of digital media, this traditional art form has become desolate. "Reconstruction" is an interactive digital artwork inspired by the production and performance of Chinese shadow puppet. The scenes and characters are designed based on the art style of shadow puppet. The participant's motion is captured with a Kinect and used to control the motion of the character.
Increasing GPU Translation Reach by Leveraging Under-Utilized On-Chip Resources

Many GPU applications issue irregular memory accesses to a very large memory footprint. We confirm observations from prior work that these irregular access patterns are severely bottlenecked by insufficient Translation Lookaside Buffer (TLB) reach, resulting in expensive page table walks. In this work, we investigate mechanisms to improve TLB reach without increasing the page size or the size of the TLB itself. Our work is based around the observation that a GPU’s instruction cache (I-cache) and Local Data Share (LDS) scratchpad memory are under-utilized in many applications, including those that suffer from poor TLB reach. We leverage this to opportunistically utilize idle capacity and port bandwidth from the GPU’s I-cache and LDS structures for address translations. We explore various potential architectural designs for each structure to optimize performance and minimize complexity. Both structures are organized as a victim cache between the L1 and L2 TLBs to boost translation reach. We find that our designs can increase performance on average by 30.1% without impacting the performance of applications that do not require additional reach.
FOXP4 differentially controls cold-induced beige adipocyte differentiation and thermogenesis

Beige adipocytes possess a discrete developmental origin and notable plasticity in thermogenic capacity in response to various environmental cues. But the transcriptional machinery controlling beige adipocyte development and thermogenesis remains largely unknown. By analyzing beige adipocyte-specific knockout mice, we identified a transcription factor, Forkhead Box P4 (FOXP4) that differentially governs beige adipocyte differentiation and activation. Depletion of Foxp4 caused a decline in the frequency of beige preadipocytes by switching their cell fate towards fibroblastic cells at the expense of beige adipocytes. However, we observed that ablation of Foxp4 in differentiated adipocytes profoundly potentiated their thermogenesis upon cold exposure. Of note, the outcome of Foxp4-deficiency on UCP1-mediated thermogenesis was confined to beige adipocytes, rather than to brown adipocytes. Taken together, we submit that FOXP4 primes beige adipocyte cell fate commitment and differentiation by potent transcriptional repression of the thermogenic program.
Fast Video Visual Quality and Resolution Improvement using SR-UNet

In this paper, we address the problem of real-time video quality enhancement, considering both frame super-resolution and compression artifact-removal. The first operation increases the sampling resolution of video frames, the second removes visual artifacts such as blurriness, noise, aliasing, or blockiness introduced by lossy compression techniques, such as JPEG encoding for single-images, or H.264/H.265 for video data. We propose to use SR-UNet, a novel network architecture based on UNet, that has been specialized for fast visual quality improvement (i.e. capable of operating in less than 40ms, to be able to operate on videos at 25FPS). We show how this network can be used in a streaming context where the content is generated live, e.g. in video calls, and how it can be optimized when video to be streamed are prepared in advance. The network can be used as a final post processing, to optimize the visual appearance of a frame before showing it to the end-user in a video player. Thus, it can be applied without any change to existing video coding and transmission pipelines. Experiments carried on standard video datasets, also considering the H.265 compression, show that the proposed approach is able to either improve visual quality metrics given a fixed bandwidth budget, or video distortion given a fixed quality goal.
TSA-Net: Tube Self-Attention Network for Action Quality Assessment

In recent years, assessing action quality from videos has attracted growing attention in computer vision community and human-computer interaction. Most existing approaches usually tackle this problem by directly migrating the model from action recognition tasks, which ignores the intrinsic differences within the feature map such as foreground and background information. To address this issue, we propose a Tube Self-Attention Network (TSA-Net) for action quality assessment (AQA). Specifically, we introduce a single object tracker into AQA and propose the Tube Self-Attention Module (TSA), which can efficiently generate rich spatio-temporal contextual information by adopting sparse feature interactions. The TSA module is embedded in existing video networks to form TSA-Net. Overall, our TSA-Net is with the following merits: 1) High computational efficiency, 2) High flexibility, and 3) The state-of-the-art performance. Extensive experiments are conducted on popular action quality assessment datasets including AQA-7 and MTL-AQA. Besides, a dataset named Fall Recognition in Figure Skating (FR-FS) is proposed to explore the basic action assessment in the figure skating scene. Our TSA-Net achieves the Spearman's Rank Correlation of 0.8476 and 0.9393 on AQA-7 and MTL-AQA, respectively, which are the new state-of-the-art results. The results on FR-FS also verify the effectiveness of the TSA-Net. The code and FR-FS dataset are publicly available at https://github.com/Shunli-Wang/TSA-Net.