Efficient and flexible visual representation learning
Creators: name: Ali, Ali
Publisher: name: RMIT University
Year: 2024
Subjects: classificationCode: 460304
subject: Computer vision
classificationCode: 460511
subject: Stream and sensor data

description: Representation learning is a fundamental problem in computer vision. Visual representation learning aims to produce effective feature vectors to train accurate machine learning models. However, existing methods require highly intensive computation and suffer from visual uncertainty, losing in-depth structures and specificity to specific shapes or objects. This research proposes novel methods to compute the accurate visual representation in multiple domains, including drone-based object tracking, image classification and few-shot learning. 

We proposed DroTrack, a high-speed drone-based object tracker. We address tracking uncertainties such as drone routing, drone motion and visual challenges such as occlusion and cluttered backgrounds. The complex drones' motion in three-dimensional space affects the tracking results of object location and scale. DroTrack produces accurate location predictions by discovering the dependency between object representation and motion characteristics. We use pre-trained Convolutional Neural Networks (CNN) to learn the object representation. Then, DroTrack leverages the geometrical relationship between the motion angle and the object scale to estimate reliable object scales. We also present a drone routing framework that resolves weather uncertainty and uses the A* algorithm and machine learning models to find the optimal drone routes. 

DroTrack utilises CNN that lose useful global structures and are affected by diverse backgrounds and unrepresentative regions. We propose novel algorithms to learn accurate visual representations, including flexible representation of images based on grid-graphs (flexgrid2vec), capturing global contextual information of images (Global Context Convolutional Networks) and Signature-Graph Neural Networks (SGNs).

flexgrid2vec is a multi-channel Graph Convolutional Network (GCN) that learns low-dimensional feature vectors of most representative image patches. Each image is represented with a graph of flexible, unique node locations and edge distances. We have investigated both spectral and non-spectral implementations of the GCN node embedding. We have implemented flexgrid2vec based on different node-aggregation methods, such as vector summation, concatenation and normalisation with eigenvector centrality. The performance of flexgrid2vec is dependent on the key-point detection method to establish the important image patches. In the following contributions, we solve this issue by presenting end-to-end visual representation networks.

We propose GCCN, a novel approach to extract global structural features to enhance the image classification. GCCN augments the convolution feature vector with a global context feature vector. This process enables CNNs to learn both local and global structural features of an image. We have extended the GCCN to solve a few-shot image classification problem. We learn metric distances between the embedded vectors or their prototype representations similar to the state-of-the-art matching and prototypical networks. However, GCCN does not fully harness the correlations among the image patches or regions.  

We propose SGN, a novel approach for visual representation learning, Signature-Graph. SGN learns latent global structures that enhance the feature representation of CNNs and constructs unique undirected graphs for each image based on the CNN feature maps. The feature maps are partitioned into a set of equal and non-overlapping patches. The graph nodes represent convolution features with the local maxima or minima in these patches. We present novel Signature-Graph designs based on horizontal and vertical edge connections between these local patches. The representation vectors are then computed based on the spectral Laplacian eigenvalues of the graphs. 

We tested our proposed methods against state-of-the-art studies. DroTrack, flexgrid2vec, GCCN and SGN show stable and outstanding results. DroTrack outperforms all the high-speed trackers and achieves comparable results with simple features compared to deep learning trackers. flexgrid2vec, GCCN and SGN outperform existing methods of recent GCNs, generative adversarial networks (GANs) and auto-encoders on image classification and few-shot image classifications.