Reviewerly/exHarmony huggingface dataset:
Representation Recovering for Self-Supervised Pre-training on Medical Images

Advances in self-supervised learning have drawn attention to developing techniques to extract effective visual representations from unlabeled images. Contrastive learning (CL) trains a model to extract consistent features by generating different views. Recent success of Masked Autoencoders (MAE) highlights the benefit of generative modeling in self-supervised learning. The generative approaches encode the input into a compact embedding and empower the model’s ability of recovering the original input. However, in our experiments, we found vanilla MAE mainly recovers coarse high level semantic information and is inadequate in recovering detailed low level information. We show that in dense downstream prediction tasks like multi-organ segmentation, directly applying MAE is not ideal. Here, we propose RepRec, a hybrid visual representation learning framework for self-supervised pre-training on large-scale unlabelled medical datasets, which takes advantage of both contrastive and generative modeling. To solve the aforementioned dilemma that MAE encounters, a convolutional encoder is pre-trained to provide low-level feature information, in a contrastive way; and a transformer encoder is pre-trained to produce high level semantic dependency, in a generative way – by recovering masked representations from the convolutional encoder. Extensive experiments on three multi-organ segmentation datasets demonstrate that our method outperforms current state-of-the-art methods.
PatchDropout: Economizing Vision Transformers Using Patch Dropout

Vision transformers have demonstrated the potential to outperform CNNs in a variety of vision tasks. But the computational and memory requirements of these models prohibit their use in many applications, especially those that depend on high-resolution images, such as medical image classification. Efforts to train ViTs more efficiently are overly complicated, necessitating architectural changes or intricate training schemes. In this work, we show that standard ViT models can be efficiently trained at high resolution by randomly dropping input image patches. This simple approach, PatchDropout, reduces FLOPs and memory by at least 50% in standard natural image datasets such as ImageNet, and those savings only increase with image size. On CSAW, a high-resolution medical dataset, we observe a 5× savings in computation and memory using PatchDropout, along with a boost in performance. For practitioners with a fixed computational or memory budget, PatchDropout makes it possible to choose image resolution, hyperparameters, or model size to get the most performance out of their model.
Interacting Hand-Object Pose Estimation via Dense Mutual Attention

2D hand-object pose estimation is the key to the success of many computer vision applications. The main focus of this task is to effectively model the interaction between the hand and an object. To this end, existing works either rely on interaction constraints in a computationally-expensive iterative optimization, or consider only a sparse correlation between sampled hand and object keypoints. In contrast, we propose a novel dense mutual attention mechanism that is able to model fine-grained dependencies between the hand and the object. Specifically, we first construct the hand and object graphs according to their mesh structures. For each hand node, we aggregate features from every object node by the learned attention and vice versa for each object node. Thanks to such dense mutual attention, our method is able to produce physically plausible poses with high quality and real-time inference speed. Extensive quantitative and qualitative experiments on large benchmark datasets show that our method outperforms state-of-the-art methods. The code is available at https://github.com/rongakowang/DenseMutualAttention.git.
PRN: Panoptic Refinement Network

Panoptic segmentation is the task of uniquely assigning every pixel in an image to either a semantic label or an individual object instance, generating a coherent and complete scene description. Many current panoptic segmentation methods, however, predict masks of semantic classes and object instances in separate branches, yielding inconsistent predictions. Moreover, because state-of-the-art panoptic segmentation models rely on box proposals, the instance masks predicted are often of low-resolution. To overcome these limitations, we propose the Panoptic Refinement Network (PRN), which takes masks from base panoptic segmentation models and refines them jointly to produce coherent results. PRN extends the offset map-based architecture of Panoptic-Deeplab with several novel ideas including a foreground mask and instance bounding box offsets, as well as coordinate convolutions for improved spatial prediction. Experimental results on COCO and Cityscapes show that PRN can significantly improve already accurate results from a variety of panoptic segmentation networks.
On Quantizing Implicit Neural Representations

The role of quantization within implicit/coordinate neural networks is still not fully understood. We note that using a canonical fixed quantization scheme during training produces poor performance at low bit-rates due to the network weight distributions changing over the course of training. In this work, we show that a non-uniform quantization of neural weights can lead to significant improvements. Specifically, we demonstrate that a clustered quantization enables improved reconstruction. Finally, by characterising a trade-off between quantization and network capacity, we demonstrate that it is possible (while memory inefficient) to reconstruct signals using binary neural networks. We demonstrate our findings experimentally on 2D image reconstruction and 3D radiance fields; and show that simple quantization methods and architecture search can achieve compression of NeRF to less than 16kb with minimal loss in performance (323x smaller than the original NeRF).
PERCEIVER-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention

We present PERCEIVER-VL, a vision-and-language framework that efficiently handles high-dimensional multi- modal inputs such as long videos and text. Powered by the iterative latent-cross-attention of Perceiver, our framework scales with linear complexity, in contrast to the quadratic complexity of self-attention used in many state-of-the-art transformer-based models. To further improve the efficiency of our framework, we also study applying LayerDrop on cross-attention layers and introduce a mixed- stream architecture for cross-modal retrieval. We evaluate PERCEIVER-VL on diverse video-text and image-text benchmarks, where PERCEIVER-VL achieves the lowest GFLOPs and latency, while maintaining competitive performance. In addition, we also provide comprehensive analyses over various aspects of our framework, including pretraining data, scalability of latent size and input size, dropping cross-attention layers at inference to reduce latency, modality aggregation strategy, positional encoding, and weight initialization strategy. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
LayerDoc: Layer-wise Extraction of Spatial Hierarchical Structure in Visually-Rich Documents

Digital documents often contain images and scanned text. Parsing such visually-rich documents is a core task for work-flow automation, but it remains challenging since most documents do not encode explicit layout information, e.g., how characters and words are grouped into boxes and ordered into larger semantic entities. Current state-of-the-art layout extraction methods are challenged by such documents as they rely on word sequences to have correct reading order and do not exploit their hierarchical structure. We propose LayerDoc, an approach that uses visual features, textual semantics, and spatial coordinates along with constraint inference to extract the hierarchical layout structure of documents in a bottom-up layer-wise fashion. LayerDoc recursively groups smaller regions into larger semantic elements in 2D to infer complex nested hierarchies. Experiments show that our approach outperforms competitive baselines by 10-15% on three diverse datasets of forms and mobile app screen layouts for the tasks of spatial region classification, higher-order group identification, layout hierarchy extraction, reading order detection, and word grouping.
Li3DeTr: A LiDAR based 3D Detection Transformer

Inspired by recent advances in vision transformers for object detection, we propose Li3DeTr, an end-to-end LiDAR based 3D Detection Transformer for autonomous driving, that inputs LiDAR point clouds and regresses 3D bounding boxes. The LiDAR local and global features are encoded using sparse convolution and multi-scale deformable attention respectively. In the decoder head, firstly, in the novel Li3DeTr cross-attention block, we link the LiDAR global features to 3D predictions leveraging the sparse set of object queries learnt from the data. Secondly, the object query interactions are formulated using multi-head self-attention. Finally, the decoder layer is repeated L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">dec</inf> number of times to refine the object queries. Inspired by DETR, we employ set-to-set loss to train the Li3DeTr network. Without bells and whistles, the Li3DeTr network achieves 61.3% mAP and 67.6% NDS surpassing the state-of-the-art methods with non-maximum suppression (NMS) on the nuScenes dataset and it also achieves competitive performance on the KITTI dataset. We also employ knowledge distillation (KD) using a teacher and student model that slightly improves the performance of our network.
Boosting neural video codecs by exploiting hierarchical redundancy

In video compression, coding efficiency is improved by reusing pixels from previously decoded frames via motion and residual compensation. We define two levels of hierarchical redundancy in video frames: 1) first-order: redundancy in pixel space, i.e., similarities in pixel values across neighboring frames, which is effectively captured using motion and residual compensation, 2) second-order: redundancy in motion and residual maps due to smooth motion in natural videos. While most of the existing neural video coding literature addresses first-order redundancy, we tackle the problem of capturing second-order redundancy in neural video codecs via predictors. We introduce generic motion and residual predictors that learn to extrapolate from previously decoded data. These predictors are lightweight, and can be employed with most neural video codecs in order to improve their rate-distortion performance. Moreover, while RGB is the dominant colorspace in neural video coding literature, we introduce general modifications for neural video codecs to embrace the YUV420 colorspace and report YUV420 results. Our experiments show that using our predictors with a well-known neural video codec leads to 38% and 34% bitrate savings in RGB and YUV420 colorspaces measured on the UVG dataset.
GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction

Many online action prediction models observe complete frames to locate and attend to informative subregions in the frames called glimpses and recognize an ongoing action based on global and local information. However, in applications with constrained resources, an agent may not be able to observe the complete frame, yet must still locate useful glimpses to predict an incomplete action based on local information only. In this paper, we develop Glimpse Transformers (GliTr), which observe only narrow glimpses at all times, thus predicting an ongoing action and the following most informative glimpse location based on the partial spatiotemporal information collected so far. In the absence of a ground truth for the optimal glimpse locations for action recognition, we train GliTr using a novel spatiotemporal consistency objective: We require GliTr to attend to the glimpses with features similar to the corresponding complete frames (i.e. spatial consistency) and the resultant class logits at time t equivalent to the ones predicted using whole frames up to t (i.e. temporal consistency). Inclusion of our proposed consistency objective yields ∼ 10% higher accuracy on the Something-Something-v2 (SSv2) dataset than the baseline cross-entropy objective. Overall, despite observing only ∼ 33% of the total area per frame, GliTr achieves 53.02% and 93.91% accuracy on the SSv2 and Jester datasets, respectively.
Learning Graph Variational Autoencoders with Constraints and Structured Priors for Conditional Indoor 3D Scene Generation

We present a graph variational autoencoder with a structured prior for generating the layout of indoor 3D scenes. Given the room type (e.g., living room or library) and the room layout (e.g., room elements such as floor and walls), our architecture generates a collection of objects (e.g., furniture items such as sofa, table and chairs) that is consistent with the room type and layout. This is a challenging problem because the generated scene needs to satisfy multiple constrains, e.g., each object should lie inside the room and two objects should not occupy the same volume. To address these challenges, we propose a deep generative model that encodes these relationships as soft constraints on an attributed graph (e.g., the nodes capture attributes of room and furniture elements, such as shape, class, pose and size, and the edges capture geometric relationships such as relative orientation). The architecture consists of a graph encoder that maps the input graph to a structured latent space, and a graph decoder that generates a furniture graph, given a latent code and the room graph. The latent space is modeled with autoregressive priors, which facilitates the generation of highly structured scenes. We also propose an efficient training procedure that combines matching and constrained learning. Experiments on the 3D-FRONT dataset show that our method produces scenes that are diverse and are adapted to the room layout.
Charting opportunities and guidelines for augmented reality in makerspaces through prototyping and co-design research

Makerspace environments are becoming popular project-based learning spaces where students interact with physical objects and peer collaboration, while developing 21st century skills and engaging with science, technology, engineering, and math (STEM) topics. At the same time, augmented reality (AR) technology, which combines physical objects with digital visualizations, is becoming increasingly applicable for makerspace activities and has potential to address challenges for student learning in makerspaces. However, there is a lack of understanding of how to use and integrate AR in real makerspace environments. In this research we use a co-design methodology to address the following questions: (1) How can AR be useful for education in makerspaces? (2) How are students impacted by the process of co-designing AR technology? and (3) What are practical considerations for integrating AR in makerspaces? We engaged in a co-design process in a semester-long makerspace course attended by 18 students in a graduate school of education. Through this process, we generated six prototypes with seven student co-designers, exploring AR use in design, fabrication, programming, electronics, and training. We also identified areas where AR technology can benefit makerspaces, such as teaching STEM skills, facilitating construction activities, enhancing contextualization of learning, and debugging. We observed that students participating in co-design demonstrated improved understanding of technology design, enthusiasm for engaging with makerspaces and AR, and increased critical thinking about AR technology. These results suggest considerations and guidelines for integrating AR technology into makerspace environments.
Content-Aware Warping for View Synthesis

Existing image-based rendering methods usually adopt depth-based image warping operation to synthesize novel views. In this paper, we reason the essential limitations of the traditional warping operation to be the limited neighborhood and only distance-based interpolation weights. To this end, we propose content-aware warping, which adaptively learns the interpolation weights for pixels of a relatively large neighborhood from their contextual information via a lightweight neural network. Based on this learnable warping module, we propose a new end-to-end learning-based framework for novel view synthesis from a set of input source views, in which two additional modules, namely confidence-based blending and feature-assistant spatial refinement, are naturally proposed to handle the occlusion issue and capture the spatial correlation among pixels of the synthesized view, respectively. Besides, we also propose a weight-smoothness loss term to regularize the network. Experimental results on light field datasets with wide baselines and multi-view datasets show that the proposed method significantly outperforms state-of-the-art methods both quantitatively and visually. The source code is publicly available at https://github.com/MantangGuo/CW4VS.
Trusting Computing as a Service for Blockchain Applications

Blockchain and smart contracts have become one of the most popular technologies to establish decentralized applications .Due to public visibility of data stored in blockchain, the information processed by a smart contract is visible to every party in a blockchain.To improve data confidentiality, this study proposes a trusted computing as a service (T CaaS) framework based on the blockchain.The framework includes a trustworthy node to perform trusted computing.Using the proposed framework, users of blockchain systems can integrate trustworthy node into blockchain systems.Users can also upload general-purposed programs and associated parameters and discover trustworthy nodes for execution via related smart contracts.The programs and parameters are encrypted so that only selected trustworthy node can decrypt the data.The execution environment returns computed results to the corresponding blockchain system.We evaluate our concept with the ESP32 microcontroller with the ATECC508A security chip and the Quorum blockchain system.Therefore, the study contributes to construct a trusted computing environment for blockchain system and preserve confidentiality at the same time.
Real-Time Tweets Streaming and Comparison Using Naïve Bayes Classifier

The rapid growth of social media sites in recent times has introduced a special environment for researching human actions. Micro blogging website (Twitter) which also allows users to access and communicate their opinions on various of topics, occurrences, brands, and service providers. Tweets have been classified into different subgroups relating to the subject of the research. Various machine learning algorithms, such as baseline, Naive Bayes classifier, support vector machine (SVM), and many others, are presently used this to categorize posts on Twitter into favourable and unfavourable groups based on their views and opinions. This paper describes a solution to improve comparison between Twitter accounts besides trying to implement Naive Bayes using perception vibrant data for training from of the Twitter database. SentiWordNet, in combined effect with Naïve Bayes classifier, can help increase tweet accuracy rate by providing hope and optimism, hatefulness, and integrity ratings for phrases in Twitter messages. Tweepy, which is a Python package and Python-Twitter APIs are used in the actual introduction of the new system. The main work of this paper is to perform real-time streaming, collect the data set while performing real time and make comparison between them using Naïve Bayes.
Design and Evaluation of Protobject: A Tool for Rapid Prototyping of Interactive Products

Designing interactive prototypes involves multiple tools and skills. In addition, several design cycles are required to iterate through idea generation, evaluation of design alternatives, and development. Consequently, prototyping tools should offer flexibility and adaptability to allow designers to quickly test and evaluate different ideas, design alternatives, materials, interactions, etc. To meet these requirements, we designed Protobject – a rapid prototyping tool aimed at making the early stages of prototyping interactive products more flexible. Protobject allows designers to reinvent and reuse existing objects for prototyping purposes by making them interactive. After introducing the features of Protobject and discussing the differences with similar tools, we present a user evaluation through two workshop sessions held in Milan during Brera Design Days and attended by 22 people. The results suggest that Protobject facilitates cooperation between people with different skills by allowing them to envision interactive prototypes together.
Residual Compensation-Based Extreme Learning Machine for MIMO-NOMA Receiver

Extreme learning machine (ELM) uses a simple machine learning (ML) architecture that allows its implementation for smart IoT network operations. However, some inaccuracy lies between the predicted and the actual data during the ELM training, which can be caused due to the limitation of the modeling representation. This paper thus investigates a residual compensation-based ELM (RC-ELM) for its application in designing a receiver for MIMO-NOMA aided IoT systems. In RC-ELM, the base ELM layers determine the relationship between the transmitted and received data and the additional layers of the RC-ELM attempt to compensate the error introduced during the training mechanism. The analysis of the appropriate number of compensation layers for training error minimization is conducted on the basis of the bit error rate (BER) and the error vector magnitude (EVM) performances of the RC-ELM training. A minimum BER improvement of 5% for user 1 and 18% for user 2 is shown with with the aid of RC-ELM for a two-IoT user instance. The EVM is marginally increased by 0.0008% for user 1 and 1.61% for user 2 in training stage. Besides, the RC-ELM receiver is also compared to the minimum mean square error (MMSE), the classic ELM and the trained multilayer perceptron (MLP) receivers in terms of BER and EVM. Both of the ELM receivers show improved performances with respect to the other receivers.
Progressive Contextual Aggregation Empowered by Pixel-Wise Confidence Scoring for Image Inpainting

Image inpainting methods leverage the similarity of adjacent pixels to create alternative content. However, as the invisible region becomes larger, the pixels completed in the deeper hole are difficult to infer from the surrounding pixel signal, which is more prone to visual artifacts. To help fill this void, we adopt an alternative progressive hole-filling scheme that hierarchically fills the corrupted region in the feature and image spaces. This technique allows us to utilize reliable contextual information of the surrounding pixels, even for large hole samples, and then gradually complete the details as the resolution increases. For a more realistic representation of the completed region, we devise a pixel-wise dense detector. By distinguishing each pixel as whether it is a masked region or not, and passing the gradient to all resolutions, the generator further enhances the potential quality of the compositing. Furthermore, the completed images at different resolutions are then merged using a proposed structure transfer module (STM) that incorporates fine-grained local and coarse-grained global interactions. In this new mechanism, each completed image at the different resolutions attends its closest composition at fine granularity adjacent image and thus can capture the global continuity by interacting both short- and long-range dependencies. By comparing our solutions qualitatively and quantitatively with state-of-the-art methods, we conclude that our model exhibits a significantly improved visual quality, even in the case of large holes.