Miracle-dz/UniFAR-FTUs huggingface dataset:
SSconv: Explicit Spectral-to-Spatial Convolution for Pansharpening

Pansharpening aims to fuse a high spatial resolution panchromatic (PAN) image and a low resolution multispectral (LR-MS) image to obtain a multispectral image with the same spatial resolution as the PAN image. Thanks to the flexible structure of convolution neural networks (CNNs), they have been successfully applied to the problem of pansharpening. However, most of the existing methods only simply feed the up-sampled LR-MS into the CNNs and ignore the spatial distortion caused by direct up-sampling. In this paper, we propose an explicit spectral-to-spatial convolution (SSconv) that aggregates spectral features into the spatial domain to perform the up-sampling operation, which can get better performance than the direct up-sampling. Furthermore, SSconv is embedded into a multiscale U-shaped convolution neural network (MUCNN) for fully utilizing the multispectral information of involved images. In particular, multiscale injection branch and mixed loss on cross-scale levels are employed to fuse pixel-wise image information. Benefiting from the distortion-free property of SSconv, the proposed MUCNN can generate state-of-the-art performance with a simple structure, both on reduced-resolution and full-resolution datasets acquired from WorldView-3 and GaoFen-2. Please find the code from the project page.
RecipeLog: Recipe Authoring App for Accurate Food Recording

Diet management is usually conducted by recording the name of foods eaten, but in fact, the nutritional value of food in the same name varies greatly from recipe to recipe. To know accurate nutritional values of the foods, recording personal recipes is effective but time-consuming. Therefore, we are developing a mobile application "RecipeLog", that assists users to write their own recipes by modifying prepared ones. In our experiments, we show that with RecipeLog users create personal recipes with 45% less edit distance compared to writing from scratch.
VIDDE: Visualizations for Helping People with COPD Interpret Dyspnea During Exercise

People with chronic obstructive pulmonary disease (COPD) experience dyspnea and dyspnea-related distress and anxiety (DDA) upon physical exertion. When performing supervised exercise, measurement of physiological data, such as heart rate (HR) and blood oxygen saturation (O2sat), are commonly used for safety, but the impacts of such monitoring on their perceptions and behaviour have not previously been studied. This paper investigates the effect of presenting live physiological data to people with COPD during exercise with a focus on its impact on perceptions of dyspnea intensity (DI) and DDA. Informed by formative interviews with 15 people with COPD, we design VIDDE, an exercise companion tool visualizing live data from a pulse oximeter, and evaluate its effect on DI and DDA through case studies involving 3 participants with COPD exercising at their homes. We also conducted design probe interviews with 6 more participants with COPD to investigate their needs and design requirements for an exercise-companion application that featured physiological data monitoring. Our results suggest that presenting live physiological data during exercise is of value, and can contribute to reduced DDA, better understanding of breathlessness sensations, while providing sufficient reassurance to encourage physical activity.
Leveraging Targeted Value Prediction to Unlock New Hardware Strength Reduction Potential

Value Prediction (VP) is a microarchitectural technique that speculatively breaks data dependencies to increase the available Instruction Level Parallelism (ILP) in general purpose processors. Despite recent proposals, VP remains expensive and has intricate interactions with several stages of the classical superscalar pipeline. In this paper, we revisit and simplify VP by leveraging the irregular distribution of the values produced during the execution of common programs. First, we demonstrate that a reasonable fraction of the performance uplift brought by a full VP infrastructure can be obtained by predicting only a few ”usual suspects” values. Furthermore, we show that doing so allows to greatly simplify VP operation as well as reduce the value predictor footprint. Lastly, we show that these Minimal and Targeted VP infrastructures conceptually enable Speculative Strength Reduction (SpSR), a rename-time optimization whereby instructions can disappear at rename in the presence of specific operand values.
Mask and Predict: Multi-step Reasoning for Scene Graph Generation

Scene Graph Generation (SGG) aims to parse the image as a set of semantics, containing objects and their relations. Currently, the SGG methods only stay at presenting the intuitive detection in the image, such as the triplet "logo on board". Intuitively, we humans can further refine these intuitive detections as rational descriptions like "flower painted on surfboard". However, most of existing methods always formulate SGG as a straightforward task, only limited by the manner of one-time prediction, which focuses on a single-pass pipeline and predicts all the semantic. Therefore, to handle this problem, we propose a novel multi-step reasoning manner for SGG. Concretely, we break SGG into two explicit learning stages, including intuitive training stage (ITS) and rational training stage (RTS). In the first stage, we follow the traditional SGG processing to detect objects and relationships, yielding an intuitive scene graph. In the second stage, we perform multi-step reasoning to refine the intuitive scene graph. For each step of reasoning, it consists of two kinds of operations: mask and predict. According to primary predictions and their confidences, we constantly select and mask the low-confidence predictions, which features are optimized and predicted again. After several iterations, all of intuitive semantics will gradually tend to be revised with high confidences, yielding a rational scene graph. Extensive experiments on Visual Genome prove the superiority of the proposed method. Additional ablation studies and visualization cases further validate its effectiveness.
Uncovering In-DRAM RowHammer Protection Mechanisms:A New Methodology, Custom RowHammer Patterns, and Implications

The RowHammer vulnerability in DRAM is a critical threat to system security. To protect against RowHammer, vendors commit to security-through-obscurity: modern DRAM chips rely on undocumented, proprietary, on-die mitigations, commonly known as Target Row Refresh (TRR). At a high level, TRR detects and refreshes potential RowHammer-victim rows, but its exact implementations are not openly disclosed. Security guarantees of TRR mechanisms cannot be easily studied due to their proprietary nature. To assess the security guarantees of recent DRAM chips, we present Uncovering TRR (U-TRR), an experimental methodology to analyze in-DRAM TRR implementations. U-TRR is based on the new observation that data retention failures in DRAM enable a side channel that leaks information on how TRR refreshes potential victim rows. U-TRR allows us to (i) understand how logical DRAM rows are laid out physically in silicon; (ii) study undocumented on-die TRR mechanisms; and (iii) combine (i) and (ii) to evaluate the RowHammer security guarantees of modern DRAM chips. We show how U-TRR allows us to craft RowHammer access patterns that successfully circumvent the TRR mechanisms employed in 45 DRAM modules of the three major DRAM vendors. We find that the DRAM modules we analyze are vulnerable to RowHammer, having bit flips in up to 99.9% of all DRAM rows.
Block Popularity Prediction for Multimedia Storage Systems Using Spatial-Temporal-Sequential Neural Networks

Predicting block popularity is of crucial importance for data placement in multi-tiered multimedia storage systems. Traditional methods, such as least recently used and exponential smoothing, are commonly employed to predict future block access frequencies and fail to achieve good performance for complex and changing access patterns. Recently, deep neural networks have brought great success to pattern recognition and prediction, which motivates us to introduce deep learning to solve the problem of block popularity prediction. In this paper, we first analyze and verify the temporal and spatial correlations among the multimedia I/O traces. Then, we design a multi-dimension feature to capture such correlations, which serves as the input of the designed deep neural network. A spatial-temporal-sequential neural network (STSNN) and its variants that capture the locality information, time dependency information, and block sequential information are proposed to predict the block popularity. We systematically evaluate our STSNN models against six baseline models from three different categories, i.e., heuristic methods, regression methods and neural network-based methods. Experiment results show that our proposed STSNN models are very promising for predicting block access frequencies under some of Huawei and Microsoft datasets and particularly achieve 2-6 times better performance compared with the baselines in terms of the I/O hit ratio, I/O recall rate and I/O prediction ratio under the Microsoft 64 MB-block dataset.
Apercevoir: Bio Internet of Things Interactive System

Apercevoir is an artwork that can perceive its environmental perturbations and convert them into a spatial sound field with location information. It consists of multiple plant cyborgs comprised of a Mimosa Pudica (sensitive plant) connected to a bioamplifier and can sense human movements by analyzing the biosignals with a machine learning model. Through sharing multiple cyborgs' biosignals, this network portrays the concept of multiple beings transcending an individual's physical confines to form a Bio Internet of Things (IOT) system capable of perception, feedback, and group decision-making within a wider scope. A particular feature of this system is its interactive bone induction headphones, where the audience can listen to a sound field including 'vibrations' of nearby human activities detected by plant cyborgs, and even warnings among the cyborg network responding to foreign disturbance and damage. This sound field invites audiences to close their eyes and listen attentively to plants while the biosignals and changes in sound reveal the presence of other entities in the space.
TIP: Time-Proportional Instruction Profiling

A fundamental part of developing software is to understand what the application spends time on. This is typically determined using a performance profiler which essentially captures how execution time is distributed across the instructions of a program. At the same time, the highly parallel execution model of modern high-performance processors means that it is difficult to reliably attribute time to instructions — resulting in performance analysis being unnecessarily challenging. In this work, we first propose the Oracle profiler which is a golden reference for performance profilers. Oracle is golden because (i) it accounts every clock cycle and every dynamic instruction, and (ii) it is time-proportional, i.e., it attributes a clock cycle to the instruction(s) that the processor exposes the latency of. We use Oracle to, for the first time, quantify the error of software-level profiling, the dispatch-tagging heuristic used in AMD IBS and Arm SPE, the Last-Committing Instruction (LCI) heuristic used in external monitors, and the Next-Committing Instruction (NCI) heuristic used in Intel PEBS, resulting in average instruction-level profile errors of 61.8%, 53.1%, 55.4%, and 9.3%, respectively. The reason for these errors is that all existing profilers have cases in which they systematically attribute execution time to instructions that are not the root cause of performance loss. To overcome this issue, we propose Time-Proportional Instruction Profiling (TIP) which combines Oracle’s time attribution policies with statistical sampling to enable practical implementation. We implement TIP within the Berkeley Out-of-Order Machine (BOOM) and find that TIP is highly accurate. More specifically, TIP’s instruction-level profile error is only 1.6% on average (maximally 5.0%) versus 9.3% on average (maximally 21.0%) for state-of-the-art NCI. TIP’s improved accuracy matters in practice, as we exemplify by using TIP to identify a performance problem in the SPEC CPU2017 benchmark Imagick that, once addressed, improves performance by 1.93 ×.
PyTorchVideo: A Deep Learning Library for Video Understanding

We introduce PyTorchVideo, an open-source deep-learning library that provides a rich set of modular, efficient, and reproducible components for a variety of video understanding tasks, including classification, detection, self-supervised learning, and low-level processing. The library covers a full stack of video understanding tools including multimodal data loading, transformations, and models that reproduce state-of-the-art performance. PyTorchVideo further supports hardware acceleration that enables real-time inference on mobile devices. The library is based on PyTorch and can be used by any training framework; for example, PyTorchLightning, PySlowFast, or Classy Vision. PyTorchVideo is available at https://pytorchvideo.org/.
Walker - An Autonomous, Interactive Walking Aid

In this paper, we describe ongoing work about a robotic walker-frame that was designed to aid patients in an orthopaedic rehabilitation clinic. The so-called Walker is able to autonomously drive to patients and then changes into a more traditional walking-frame, i.e. one that has to be pushed by the patient, but it can still help by giving navigation instructions. Walker was designed with a multi-modal user interface in such a way that it can also be used by visually, hearing or speaking impaired people.
Reproducibility Companion Paper: Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework

In this companion paper, we provide details of the artifacts to support the replication of "Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework", which was presented at MM'20. The Inter-intra Contrastive (IIC) framework aims to extract more discriminative temporal information by extending intra-negative samples in contrastive self-supervised learning. In this paper, we first summarize our contribution. Then we explain the file structure of the source code and detailed settings. Since our proposal is a framework which contain a lot of different settings, we provide some custom settings to help other researchers to use our methods easily. The source code is available at https://github.com/BestJuly/IIC.
Neural-based Rendering and Application

Rendering plays an important role in many fields such as virtual reality and film, but the high dependence on computing sources and human experience hinders its application. With the development of deep learning, neural rendering has attracted much attention due to its impressive performance and efficiency than traditional rendering. In this paper, we mainly introduce two neural rendering works, one is rendering simulation and the other is image-based novel view rendering. Moreover, we also discuss the potential applications (i.e. data augmentation) based on the results of neural rendering, which has received little attention.
DFR-Net: A Novel Multi-Task Learning Network for Real-Time Multi-Instrument Segmentation

In computer-assisted vascular surgery, real-time multi-instrument segmentation serves as a pre-requisite step. However, a large amount of effort has been dedicated to single-instrument rather than multi-instrument in computer-assisted intervention research to this day. To fill the overlooked gap, this study introduces a Light-Weight Deep Feature Refinement Network (DFR-Net) based on multi-task learning for real-time multi-instrument segmentation. In this network, the proposed feature refinement module (FRM) can capture long-term dependencies while retaining precise positional information, which helps model locate the foreground objects of interest. The designed channel calibration module (CCM) can re-calibrate fusion weights of multi-level features, which helps model balance the importance of semantic information and appearance information. Besides, the connectivity loss function is developed to address fractures in the wire-like structure segmentation results. Extensive experiments on two different types of datasets consistently demonstrate that DFR-Net can achieve state-of-the-art segmentation performance while meeting the real-time requirements.
TRiM: Enhancing Processor-Memory Interfaces with Scalable Tensor Reduction in Memory

Personalized recommendation systems are gaining significant traction due to their industrial importance. An important building block of recommendation systems consists of the embedding layers, which exhibit a highly memory-intensive characteristic. A fundamental primitive of embedding layers is the embedding vector gathers followed by vector reductions, exhibiting low arithmetic intensity and becoming bottlenecked by the memory throughput. To tackle such a challenge, recent proposals employ a near-data processing (NDP) solution at the DRAM rank-level, achieving impressive performance speedups. We observe that prior rank-level-parallelism-based NDP solutions leave significant performance potential on the table as they do not fully reap the abundant transfer throughput inherent in DRAM datapaths. We propose TRiM, an NDP architecture for accelerating recommendation systems. Based on the observation that the DRAM datapath has a hierarchical tree structure, TRiM augments the DRAM datapath with “in-DRAM” reduction units at the DDR4/5 rank/bank-group/bank level. We modify the interface of DRAM to provide commands effectively to multiple reduction units running in parallel. We also propose a host-side architecture with hot embedding-vector replication to alleviate the load imbalance that arises across the reduction units. An optimal TRiM design based on DDR5 achieves up to a 7.7 × and 3.9 × speedup and reduces by 55% and 50% the energy consumption of the embedding vector gather and reduction over the baseline and the state-of-the-art NDP architecture with minimal area overhead equivalent to 2.66% of DRAM chips.
UC-Check: Characterizing Micro-operation Caches in x86 Processors and Implications in Security and Performance

The modern x86 processor (e.g., Intel, AMD) translates CISC-style x86 instructions to RISC-style micro operations (uops) as RISC pipelines are more efficient than CISC pipelines. However, this x86 decoding process requires complex hardware logic (i.e., x86 decoder) to identify variable-length x86 instructions, which incurs high translation overhead. To avoid this overhead, the x86 processors adopt a micro-operation cache (uop cache) to bypass the expensive x86 decoder by caching the decoded uops. In this paper, we find out modern uop caches suffer from (1) security vulnerability and (2) severe cache contention between co-located SMT cores. To understand these security and performance implications of the uop cache, we propose UC-Check to extract various undisclosed features by using carefully designed microbenchmarks. With the extracted features, (1) we present two attack scenarios exploiting the uop cache as a new timing side-channel and propose a secure architecture to mitigate these attacks with negligible overhead. In addition, (2) we propose a logical uop cache allocation technique to alleviate the cache contention problem. For the evaluation, we extract many undocumented features on a wide spectrum of modern x86 processors and show that our proposed schemes (e.g., security attack/defense, performance optimization) are directly applicable to commodity x86 processors. For example, our logical uop cache allocation improves uop cache hit ratios by up to 1.33 × and achieves up to 1.04 × throughput improvement. We release all software artifacts (e.g., microbenchmarks used for feature extraction, attack proof-of-concept codes, logical uop cache allocation) to the community so that the users can easily reproduce our results and gain insights for further research.
SmartEye: An Open Source Framework for Real-Time Video Analytics with Edge-Cloud Collaboration

Video analytics with Deep Neural Networks (DNNs) empowers many vision-based applications. However, deploying DNN models for video analytics services must address the challenges of computational capacity, service delay, and cost. Leveraging the edge-cloud collaboration to address these problems has become a growing trend. This paper provides the multimedia research community with an open source framework named SmartEye for real-time video analytics by leveraging the edge-cloud collaboration. The system consists of 1) an edge layer which enables video preprocessing, model selection, on-edge inference, and task offloading; 2) a request forwarding layer which serves as a gateway of the cloud and forwards the offloaded tasks to backend workers; and 3) a backend worker layer that processes the offloaded tasks with specified DNN models. One can easily customize the policies for preprocessing, offloading, model selection, and request forwarding. The framework can facilitate research and development in this field. The project is released as an open source project on GitHub at https://github.com/MSNLAB/SmartEye.
aBio: Active Bi-Olfactory Display Using Subwoofers for Virtual Reality

Including olfactory cues in virtual reality (VR) would enhance user immersion in the virtual environment, and precise control of smell would facilitate a more realistic experience for users. In this paper, we present aBio, an active bi-olfactory display system that delivers scents precisely to specific locations rather than diffusing scented air into the atmosphere. aBio provides users with a natural olfactory experience in free air by colliding two vortex rings launched from dual speaker-based vortex generators, which also has the effect of cushioning the force of air impact. According to the various requests of different applications, the collision point of the vortex rings can be positioned anywhere in front of the user's nose. To verify the effectiveness of our device and understand user sensations when using different parameters in our system, we conduct a series of experiments and user studies. The results show that the proposed system is effective in the sense that users perceive smell without sensible haptic disturbance while the system consumes only a very small amount of fragrant essential oil. We believe that aBio has great potential for increasing the level of presence in VR by delivering smells with high efficiency.