# Contextual Coefficients Excitation Feature: Focal Visual Representation for Relationship Detection
## 3 Methods
### CCEF: Focal Visual Representation  
First, ours proposed \(R_{i}\) is the focal visual representation, which adaptively focus on the local dimensions of each object viusal features for relationships. It is constrained by the label and spatial information to boost the discriminability of representation. \(R_{i}\) is defined as follows:  
\[R_{i}=\mathbf{F}(M_{i},V_{i}), \tag{1}\]  
where \(\mathbf{F}(\cdot)\) refers to the feature-wise excitation and original visual features \(V_{i}\) are obtained by feeding image regions of object to the feature extractor [2], as in Figure 2.  
**Contextual Coefficients. ** The contextual coefficient \(M_{i}\) indicates the importance of each object visual features which is based on label and spatial descriptors, designed as:  
\[M_{i}=G_{i}\otimes F_{i}, \tag{2}\]  
where \(G_{i}\) is the **generated-label coefficient** from the object label with the conditional WGAN and \(F_{i}\) is the **spatial coefficient**. \(\otimes\) denots the elements-wise multiplication to ensure that multiple dimensions are allowed to be emphasised opposed to one-hot activation. so as in Figure 2.  
**Generated-label coefficients. ** The generated-label coefficient \(G_{i}\), as a part of importance descriptor, is relationship-related semantics. It is mapped from pure semantic feature space to relationship-related space by conditional WGAN and defined as:  
\[G_{i}=g(z_{i},C(y_{i})), \tag{3}\]  
where function \(g\) (parameterized by \(\theta_{g}\)) is the generator of conditional WGAN, \(z_{i}\in\mathbb{R}^{d_{z}}\) is a random noise vector sampled from a multidimensional centered Gaussian and \(C(y_{i})\) is the condition vector to direct the coefficients generation process. Besides, the discriminator \(d\) (parameterized by \(\theta_{d}\)) tries to distinguish whether generated-label coefficient can represent label coefficient with the conditional vector \(C(y_{i})\) or not. The components \(d\) and \(g\) iteratively play the two-player minimax game with the objective function, where \(d\) tries to maximize the GAN loss and \(g\) tries to minimizes it. The structure of our conditional WGAN is shown in Figure 3.  
Figure 2: Overview of our visual relationship detection framework. A set of detected object regions are first cropped from input image and fed into feature extractor, a base CNN network, to extract the visual features. Then, CCEF is produced by exciting visual features rely on contextual coefficients, which are carried out by the generated-label coefficients and spatial coefficients. The relationship are represented by subtracting objects from subjects. **Spatial coefficients. ** The spatial coefficient \(F_{i}\) is the embedding of object spatial information, defined as follows:  
\[F_{i}=r(S_{i}|\theta_{s}), \tag{4}\]  
where \(r(\cdot)\) is the Fully Connected layers (FC) layer with ReLU activation, \(\theta_{s}\) is the parameter of FC layer. Specifically, \(S_{i}\) is the spatial feature from the bounding boxes of object pairs, similar to the ones in [16]:  
\[\left[\frac{x_{i}-x_{j}}{w_{j}}\,\frac{y_{i}-y_{j}}{h_{j}},log\frac{w_{i}}{w_{j}}, log\frac{h_{i}}{h_{j}}\right], \tag{5}\]  
where \((x_{i},y_{i},w_{i},h_{i})\) and \((x_{j},y_{j},w_{j},h_{j})\) are bounding boxes of candidated object pairs \(B_{i}\) and \(B_{j}\). Note that subscript \(i\), \(j\) indicate different object instances. **Relationship Representation. ** Referring to the TransE [16; 35], the relationship representation \(R_{rel}\) is modeled as the translation vector of object reprensatation pairs(\(R_{i}\) and \(R_{j}\)) by mapping them to the relation space. It is defined as follows:  
\[R_{rel}=R_{i}\odot R_{j}, \tag{6}\]  
where \(\odot\) denotes element-wise subtraction, so as in Figure 2.0  
Although focal visual representation encodes the appearance of both objects, it is difficult to directly model spatial correlation between predicates and objects with pixel values of image.