# Zero-Shot Recognition Enhancement by Distance-Weighted Contextual Inference
## 3 Method
### Model Overview  
Traditional ZSR methods classify an unseen object using only its individual visual information. On the contrary, our proposed model aims to classify an unseen object with the help of contextual information obtained from its surroundings in a multi-object scene. We define potentially related surrounding objects under the following assumptions; (1) they represent non-unseen classes and are not the targets of our recognition task; (2) they are detected by a pre-trained object detector or classifier before the ZSR of the target object; and (3) each one includes predicted candidates and corresponding prediction probabilities. Consequently, the surrounding information (\(SI\)) of an unseen image feature \(i\) with multiple surrounding objects is specified by the following equations,  
\[SI(i)=\{s_{ij}\}_{j=1}^{web}, \tag{1}\]\[si=(\{(c_{k},p_{k})\}_{k=1}^{N^{cond}},d), \tag{2}\]  
where \(si\) denotes the surrounding information for a single surrounding object and \(SI\) consists of multiple \(si\)s with the number, \(N^{sobj}\). \(c_{k}\) and \(p_{k}\) are the class label of the \(k\)-th predicted candidate of a surrounding object and the prediction probability of the corresponding candidate \(c_{k}\), respectively. In particular, \(c_{k}\in S\), and \(p_{k}\) is a soft-max value of the prediction score among \(N^{cond}\) candidates. Our proposed model takes an unseen image feature \(i\) and its surrounding information \(SI(i)\) as its inputs and predicts the most likely class label \(c\) as its output. Specifically, let the aforementioned classifying function be \(f:I\to E\) between a visual feature space \(I\in\mathbb{R}^{d_{i}}\) and a semantic embedding space \(E\in\mathbb{R}^{d_{e}}\) for class labels. The classifying function \(f\) outputs the semantic embedding vector \(e\) that maximizes the scoring function \(F\) as follows and our model finally produces a prediction for the corresponding class label \(c\) of the output semantic embedding vector. \[f(i,SI(i))=\arg\max_{e\in E}F(i,e,SI(i)). \tag{3}\]  
For the process of ZSR, as shown in Figure 1, our model acts on two collaborating branches, the instance-level visual inference of an unseen object and the contextual inference with surrounding information. The visual inference basically follows the process of traditional ZSL models: the extraction of a feature vector from a target unseen image and the prediction of a visual score for each target class label using a zero-shot classifier. The contextual inference, a novel process, performs similarity measurements between each of target class labels and visual candidates of the surrounding objects to obtain a contextual score.Similarities are measured using a knowledge graph and/or a semantic embedding space. The prediction of the visual inference is calibrated and advanced by the result of the contextual inference. More detailed processes and formulations are described in Sections 3.3 and 3.4. Consequently, the scoring function \(F\) for a specific class label can be specified by combining the instance-level visual inference function \(G\) and the contextual inference function \(H\) as follows, where \(\alpha\) is a balancing factor for an usage ratio between the visual and contextual scores:  
\[F(i,e,SI(i))=\alpha\cdot G(i,e)+(1-\alpha)\cdot H(e,SI(i)). \tag{4}\]  
As shown in the intuitive and assumptive example in Figure 1, an office environment contains several objects including the target unseen object (orange bounding box), a monitor, and its surrounding  
Figure 1: The architecture of our proposed model with an example in an office environment. The model performs Zero-Shot Recognition (ZSR) on two main branches: the instance-level visual inference and the contextual inference.The visual inference first infers the class label of the extracted feature vector \(i_{1}\) of the target object, which is a process used by existing methods.