Supplementary Material: Fully Convolutional Scene Graph Generation

This supplementary document is organized as follows: 1) more training details in Section 1; 2) details on network inference in Section 2; 3) detailed architectures in Section 3; 4) qualitative studies in Section 4. All notations are followed from the paper. 1. Training We defined the center heatmaps in Equation 1 from the paper such that the centers are converted into 2D Gaussian masks. The standard deviations σx and σy are computed based on the desired radii a and b along x-axis and y-axis, respectively: (σx, σy) = 1 3 (a, b) = 1 3 b( √ 2− 1) s/τ + 1c. (1) The values of a and b are determined by Eq.(1) such that for any point within the ellipse region of radii a and b, when using it as a center to create a bounding box of object size s, the bounding box has at least 0.5 intersection over union (IoU) with the GT bounding box. Then the prediction of center heatmaps can be supervised by forms of distance losses such as Gaussian focal loss [5, 7, 13] with hyper-parameters α = 2 and γ = 4 for weight balancing. Let Ô be the predicted center heatmap, then the pixel-wise loss LOc,x,y is defined as: LOc,x,y = { (1− Ôc,x,y) log(Ôc,x,y) if Oc,x,y = 1 (Ôc,x,y) (1−Oc,x,y) log(1− Ôc,x,y) o/w. (2) 2. Inference In this section, we provide more details on inference and post-processing of our proposed fully convolutional scene *Work done in part as an intern at Futurewei Technologies Inc. graph generation (FCSGG). Our model outputs four dense feature maps: center heatmaps Ô, center offsets ∆̂ , object sizes Ŝ and relation affinity fields (RAFs) F̂. To get the object centers, we follow the same step in [13]. Specifically, a in-place sigmoid function is applied to the predicted center heatmaps Ô such that their values are mapped into the range of [0, 1]. Then a 3 × 3 max pooling is applied to center heatmaps for filtering duplicate detections. For a point p = (x, y), the value of Ôc,x,y is considered as the measurement of the center detection score for object class c. Then peaks in center heatmaps are extract for each object class independently. We keep the top 100 peaks by their scores and get a set of object centers {o} i=1 with object classes {c} i=1. To get the corresponding center offset and object size given a detected object center ô = (x̂, ŷ), we simply gather the values from ∆̂ and Ŝ at ô. We can get the center offset δ̂ i = ∆̂x̂i,ŷi = (δ̂ i x, δ̂ i y), and the object size ŝ = Ŝx̂i,ŷi = (ŵ x, ĥ i y). Finally, the bounding box (x̂0, ŷ i 0, x̂ i 1, ŷ i 1) of object b̂ i can be recovered by x̂0 = ( x̂ + δ̂ x − ŵ/2 ) · τ ŷ 0 = ( ŷ + δ̂ y − ĥ/2 ) · τ x̂1 = ( x̂ + δ̂ x + ŵ /2 ) · τ ŷ 1 = ( ŷ + δ̂ y + ĥ /2 ) · τ, (3) where τ is the stride of the output features. As for multi-scale prediction, we gather the top 100 detected objects for each scale, then perform a per-class nonmaximum suppression (NMS) and keep the top 100 boxes from all the detections, e.g., if there are 5 scales, we will keep the top 100 boxes from the 500 boxes across all the five scales. 2.1. Path Integral For multi-scale RAFs, we select the valid object pairs from the kept 100 detections {o} i=1 following the rule defined in Section 4.2 from the paper. For example, we define