Object-Centric Relational Representations for Image Generation
Authors: Luca Butera
Andrea Cini
Alberto Ferrante
Cesare Alippi
Venue: Accepted by TMLR ()

Abstract: Conditioning image generation on specific features of the desired output is a key ingredient of modern generative models. However, existing approaches lack a general and unified way of representing structural and semantic conditioning at diverse granularity levels. This paper explores a novel method to condition image generation, based on object-centric relational representations. In particular, we propose a methodology to condition the generation of objects in an image on the attributed graph representing their structure and the associated semantic information. We show that such architectural biases entail properties that facilitate the manipulation and conditioning of the generative process and allow for regularizing the training procedure. The proposed conditioning framework is implemented by means of a neural network that learns to generate a 2D, multi-channel, layout mask of the objects, which can be used as a soft inductive bias in the downstream generative task. To do so, we leverage both 2D and graph convolutional operators. We also propose a novel benchmark for image generation consisting of a synthetic dataset of images paired with their relational representation. Empirical results show that the proposed approach compares favorably against relevant baselines.

Published in Transactions on Machine Learning Research (07 (07/2024)
Object-Centric Relational Representations for Image Generation
Luca Butera
fuca.butera@uuce Gusii
The Swiss AT Lab IDSIA 8 Università della Svizzera italiana
Andrea Cini
undrea.cinisSusti ch
The Swiss AI Lab IDSIA & Università della Svizzera italiana
Alberto Ferrante
alberto. Siierto.eerronteeuss.co
The Swriss Al Lab IDSIA & Università della Svizzera staliana
Cesare Alippi
cesare.alippi@usi.co ch
The Swiss AI Lab IDSIA & Università della Svizzera italiana
Politecnico di Milano
Reviewed on OpenReview: https: openreview. net/ forum? id- BBN999
Abstract
Conditioning image generation on specific features of the desired output is a key ingredient of modern generative models. However, existing approaches lack a general and unified way of representing structural and semantic conditioning at diverse granularity levels. This paper explores a novel method to condition image generation, based on object-centric relational representations. In particular, objects in image on the altributed graph representing their structure and the associated semantic information. show that such architectural biases entail properties that facilitate the manipulation and conditioning of the generative process and allow for regularizing the training procedure. The proposed conditioning framework implereented by means of a neural network that learns to generate a 2D, multi-channel, layout mask of the objects, which can be used soft inductive bias in the downstream generative task. To do so, we leverage both 2D and graph convolutional operators. We also propose novel benchmark for image generation consisting of synthetic dataset of images paired with their relational representation. Empirical results show that the proposed approach compares favorably against relevant baselines.
propose methodology to condition the generation of
1 Introduction
Graphs are an abstraction that allows for representing objects as collections of entities and binary relation- ships. Previous research on graph-based image generation has been limited to the high-level conditioning of the image content by means of scene graphs, i.c., graphs where nodes represent objects, and edges denote prbject-predicate--ojedd relationships (Johnson et al. 2018 [vgi al. 2021). Conversely, conditioning on desired fine-graine -reegganned properties, e.g., spatial location, arrangement or visual attributes, is usually performed without considering relational structure. In fact, most of the literature dealing with pose-constrained image generation, c.g., (Ma et al. 2017; Siarohin et al.) 2018 Neverova et al. 2018 Qian et al., 2018 Jiang et al. 2022), represents key points and semantic attributes with 2D masks or feature vectors; hence not ex- ploiting known relationships among the components of the object being generated. Indeed such approaches are usually limited to conditioning the generation on fixed template structure and lack ficxibility. This paper aims at taking full advantage of graph representations this context; our approach fills the above research gap and addresses the shortcomings of existing methods by encoding relational inductive biases into

---

Published in Transactions on Machine Learning Research (07/2024)
the processing. In our method, graphs used both effective and compact, fine-grained object-centric representations of the image content as well as an architectural bias for the neural architecture. We call such representations attributed pose graphs, i.e. graphs whose nodes can represent particular landmarks in an object's structure and have both positional (location) and, possibly, semantic attributes (c.g., color, class). Edges, instead, account for relationships among nodes, capturing the object's morphology. Our formulation allows for encoding all the desired properties of the generated image in such a graph, without relying on additional inputs, such as reference images or similar, allowing for high flexibility in specifying the condition- ing. The generation of the scene can then be manipulated, without any change to the model architecture, by editing the graph, e.g., by modifying the nodes' attributes and/or the number of landmarks used to represent each object. This is thanks to the exploitation of newral message passing [Gilmer et al. 2017 and gruph newral networks (GNNs) [Bacciu et al. 2020: Bronstein et al. 2021 used for constraining the processing.
Our model, named GraPhOSE, relies on learning a multi-channel layout mask from the structured repre- sentation of the scene. is trained jointly with a downstream model (image docoder) that contextually exploits such mask to constrain the generative process. To overcome the lack of pre-annotated object masks for specific use cases, we also propose the usage of surrogate masks corresponding to procedurally generated synthetic pose graphs, as targets for a supervised pro-training of GraPhOSE. After the pertaining, the full architecture, i.e., GraPhOSE and the downstream model, can be trained end-to-end on the task at hand. Additionally, since pre-training relies on randomly generated graphs potentially (partially) outside the target distribution, the proposed method can act as a regularization, preventing overfitting the most common poses in the target dataset.
To the best of our knowledge, this is the first work to use object-centric graph-based relational representa- tions to condition a neural generative model. Furthermore, complement the methodological contributions by introducing benchmark for pose-conditioned image generation: we propose synthetic dataset named Tose-Representable Objects (PRO), which aims at assessing the performance of generative models in condi- tioning image generations on fine-grained structural and semantic information of the objects in the scene. PRO consists of images containing stylized 2D objects that can be rendered starting from a relational rep- resentation, ie., graph, encoding their structure and style. Our contribution can be then summarized as follows:
We provide novel and general methodology to solve the task of generating images conditioned on an attributed graph that represents the structure and semantics of objects in the scene.
We provide a specific implementation of such methodology, logether with a learning procedure based on a task-independent pre-training to enable transfer to different problems.
We propose novel benchmark for image generation conditioned on relational representations of the objects in the scene.
Experimental results show that our approach compares favorably against non-relational baselines and that the proposed method is flexible and can be applied to different settings.
2 Preliminaries
A pose graph is couple G (V, V is the set of vertices (or nodes) and E is the set of edges that connect such nodes. We define node attributes v; (p;, æx) where p; R2, x, Rd represents the 2D position of the i-th node in V and its d-dimensional feature vector, respectively. We denote with V the node attribute matrix V = (P stacking all the node attribute vectors. The edge connecting the i-th to the j-th node is indicated as eij (i.j). We assumne the graph to be undirected and indicate its (symmetric) adjacency matrix as A; nonetheless, our approach can be seamlessly extended to directed graphs. The described graph can be processed by stacking GNN layers; in particular, we rely on the message passing framework (Gilmer let al. (201)), which provides template for building GNNs by specifying update and aggregation functions to process node's representation and those of its neighbors.
2

---

Published in Transactions on Machine Learning Research (07 (024)
{
Surrogate Pre-Trained
GraPhOSE
Mask Generator Hg
Downstream Network
4
Layout Mask
Encoder
{x, P
Generated Image
Figure Our pipeline, with GraPhOSE in grey. Po gets pre-trained on surrogate masks. The downstrcam model, in yellow, can be any trainable generative model that accepts a 3-d tensor as conditioning input. The whole pipeline can be trained end-to-end.
0.2
08
0.
0.6
1.0
0.0
0.
0.-
0.4
0.0
1.0
1.0
Figure Surrogate mask for a random graph (left) and for a graph representing a person (right). Node positions in graph space are normalized between 0 and 1.
We indicate deep generative model neural network that, optionally given conditioning input, be trained to match desired data distribution. The sampling is often obtained by feeding random noise from fixed distribution to the model, while conditioning allows to control the generalive process, e.g., by specifying class label. In our case, we consider generative processes conditioned on the objects in the scene, represented as attributed pose graphs.
3
Graph-based Object-Centric Generation
In this Section, we first provide a high-level description of our method, consequently propose an archilecture implementing the framework and finally present our surrogate pre-training objective.
3.1 Overview
Figure provides high-level view of our approach. Given an input pose graph G, we wish to output layout mask E [0, 1 xx which will then be used to condition the downstream model on a target generative task. The mask generation process designed to obtain the final mask by aggregating masks localized w.r.t. each node. Tn particular, the generation of decomposed into the generation of mask M, [0, 1) HxW and a feature vector fi E [0, 1) for cach node v, such that:
L- EsoM. M
where & indicates the tensor product. Notably, each node mask, M,, is dependant on pose attributes, P. only, while features f, depend both pose and attributes (PIX), that the former encodes only
3

---

Published in Transactions on Machine Learning Research (07/2024)
structural information, while the latter will also eventually account for semantics. In particular, both fa and M; are computed by means of two learnable mappings, such that the core processing of GraPhOSE can be summarized as:
F 2 (V,A)
M p4 A)
(2)
Σ V
1
fr 2 M,
EEV
where A is the graph's adjacency matrix, M and F indicate respectively the tensor encompassing all the M; masks and the matrix having as rows the node level representations f.. The & function can be learned end-to-end with the downstream model, as shown in Section 55) Differently, No is pre-trained on a surrogate task designed to foster the learning of masks coherent with the structure of the object being generated. After pro-training, Hd can be fine-tuned by being trained end-to-end with the downstream model on the larget lask. This approach is suitable to flexibly condition the generation of different objects based on their structure. Notably, we implement our method with two neural networks that operate in parallel: the encoder (.)) and the mask generator (Ue). The former implements function & from Equation (2), while the latter implements function μag. The outputs of the two networks combined as in Equation 2 to obtain the layout mask.
3.2 Implementing the encoder 4
The encoder (o)) network is based on what we call a pose convolution layer, which consists of the following messago-passing paaaggg operator:
h = (9s (h p.) + Σ 9i (h,, P, pi),
JEN(()
where N (i) the set of neighbours of the i-th node, while gr. 9i and 9g can be any learnable function (e.g., MLPs). Consistently with the previous naming convention, h; and p; indicate node features and position respectively. The layer is inspired by PointNet (Qi et al. 2017), but uses two distinct operators for processing the representation of the central node and computing messages coming from neighbors. In particular, 9s be seen as implementing a parametrized skip connection to mitigate over-smoothing node representations (Li ct al. 2019 The final node encodings are obtained by stacking blocks of the form
GHIP BN (RELU (PCoNV (GHIP)))
which is common way of chaining processing layers in deep neural networks. In particular, BN denotes batch normalization (Toffe Szegedy 2015), RELU is the activation function, PCONV our pose convolution layer and GHIP a pose graph with features H and node coordinates P.
3.3 Implementing the mask generator P40
The mask generator (Ho) network consists of a first block stacked pose convolutional layers analogous to the ones Equation 4 The outputs these layers are then reshaped into bi-dimensional matrices used as input for the second stack of layers consisting of a combination shared 2D convolution blocks, interleaved with relational pose convolution 2D layers. Such layers implement the following message-passing function:
O.
Σ go (H,, H,) JEN()
(5)
H 9s H (O.) ④ 0g (0.)
where, denotes the Hadamard product, (i) is the set of neighbors of node i and 9s. Si and Jy can be any learnable function that has two-dimensional inputs and outputs.
4

---

Published in Transactions on Machine Learning Research (07/2024)
In case, 99 is 2D convolutional layer while 9o is convolutional block with upsampling and skip- connection and Os is linear upsampling operation followed by 2D convolution, where the upsampling is needed to match the dimensions of the output of Jo. The rationale behind the design of Equation (5) is promoting heterogencity between the masks generated by different nodes. Notably, Is can act as a skip connection preserving the node representations while the galing operation allows for selectively aggregal- ing information coming from neighbors. Indeed, over-smoothing the node features would be particularly detrimental as it would compromise the locality of the masks learned w.r.t. each node. Said property is desirable as, per Equation { this would in turn make the learned node features, fi, localized w.r.t. their spatial neighborhood. Note that these soft constraints, i.c., architectural biases, can be seen as form of regularization aligned with object-centric generation tasks.
3.4 Surrogate Task
As mentioned previously, we want to pro-train GraPhOSE, in particular μø, on the generic surrogate task of mapping a pose graph to a 2D mask. For this purpose, we define the surrogale mask associated with a pose graph G, as shown in Figure which, intuitively, is a grayscale image that depicts the structure of the graph.
The mask for the whole graph is obtained by composing partial masks relative to each node and edge. In particular, given pixel coordinates e, we define the value of each i-th partial surrogate mask at that pixel
SN.; (c) = exp (-
Ip: cl2
cxx
(6)
202
and, analogously, the value of the partial surrogate mask associated with each eij edge as
exp ( (-di((c) TTJ² dej(())
SE (c)
(7)
(2T)² del (Tij)
where Tiy denotes a 2 2 rotation and scaling matrix dependent on the length and orientation of the segment connecting the i-th and j-th nodes (see the supplementary material for the details). Conversely, diş is defined as
de(e)=c-(Pitb +pj Pj
pi
dy(c)=
(8)
and denotes the distance between e and the midpoint between the coordinates of nodes i and j.
The final surrogate mask is obtained, for each pixel c, as
Sg (c) = Σ (c)+
SE, (c)
(9)
EG
(1))
which is the pixel-wise sum of the masks associated with each node and edge. All the values are then clipped between 0 and 1. More details about the computation of the mask can be found in Appendix A.3 Intuitively, the mask corresponding to node is obtained by considering an isotropic bi-variate gaussian centered into the node's coordinates and with standard deviation a. The mask corresponding to an edge, instead, is obtained by considering a bi-variate gaussian centered w.r.t. the midpoint between the edge's vertices, and a covariance matrix dependent on the distance between the two points and the orientation the line joining them. The surrogate mask obtained such a way is agnostic w.r.t. the object represented by the graph and hence general. Note that differently from, e.g., segmentation masks, the mask we are referring to depends entirely on the structure of the objects in the image being generated and not on their class.
As final remark, the surrogate mask has a lattice structure, which may not properly resemble the desired mask for all larget lasks; indeed, depending on the specific object, some loops should be filled and the silhouette of each specific part refined. Nonetheless, such surrogate masks are helpful in providing supervision for the pre-training routine and act as a regularization for the model. Details on the finc-tuning procodure on downstream tasks are discussed in Section 6.1 The pre-training routine is carried out by minimizing a surrogate loss Lye based reconstruction error, e.g., mean squared error binary cross entropy.
5

---

Published in Transactions on Machine Learning Research (07/2024)
4 Related Works
Image generation is popular application of deep learning models from generalice adversarial networks
(GANs) (Goodfellow et al. 2014;
Gui et al. 2021), to variational auloencoders (VAFs) (Kingma & Welling Park et al. 2020) and, more recently, diffusson models [Sohl-Dickstein et al. Song & Ermon 2019 Ho et al. 2020). Concurrently, many researchers have explored ways of con-
2014
Vahdat & Kautz, 2020
2015
ditioning the generation of such images, from simple class labels (Mirza & Osindero 2014 Brook et al.
2019),
to fully articulated sentences (Ramesh et al., 2022; Rombach et al. 2022) or even other images (Isola
et al. 2017 Liu et al., 2017 Zhu et al., 2017). Although no previous work directly exploits pose graphs to guide image generation, several approaches focused on scene-gran ceene-gruh--ond pruph-conditioncl image generation and pose- conditioned image generation. Even though these tasks present some similarities with our work, they are fundamentally different, as discussed in the remainder of this section.
Scene-Graph- Chana--Gaaa--ooiiiionn Image Generation Scene graphs are a way of representing visual scene context as graph linking object nodes through predicate edges. These graphs are obtained by parsing natural language sentences (e-g.: the phrase A man sils on bench turns into a graph with two nodes with attributes man and bench, respectively, linked by an edge with attribute to sit), making these approaches similar to text-based image gencration. The first mothod for conditioning image generation on scone graphs was introduced by Johnson et al. (2018), which propose a GNN mapping the scene graph inlo a scene layout used to bias a generator network. [vgi et al.] (2021) improved such an approach by designing a GNN that directly operates on 2D feature maps. Differently from the above methods, however, our formulation represents objects graphs of related parts, with both structural and semantic information; scene graphs contain only semantics and represent one object per node, as they are derived from natural language parsing. This makes such methods unsuitable for our problem setting.
Pose- CossiCondii Image Generation Several prior works condition image generation on the pose of the objects being generated. In particular, several methods deal with images from human poses, with Ma et al. 2017) introducing the first deep learning approach based on conditioning reference image on target pose. The method introduced by Ma et al.] (2017) has been thereafter extended by implementing deformable convolutions (Siarohin et al. 2018), adopting dense poses (Neverova et al. 2018) and global style altributes (Men et al.] 2020) as conditioning. Moreover, it has been used to address the problem of person re-identification (Qian et al. 2018). Horiuchi et al.] (2021) exploits a graph-based representation for the pose paired with a reference image encoding semantics. To the best of our knowledge, none of the previous works jointly encode pose and semantics information in a graph and then use such graph
architectural bias for the neural processing. Exploiting this previously unused relational information allows our method to compare favorably against the state of the art w.r.t. flexibility, generality, and effectiveness, Moreover, note that this family of techniques solves an image-to-image task, conditioned on the pose (i.e., given a person's 's image as imput, the model outputs an image with the same context, where the person matches the target pose). This is a very different problem w.r.t. to the graph-to-image task we are tackling. While a direct comparison would not be meaningful, classical pose-conditioned approaches can be better suited to tasks where the conditioning is template image that needs to be modified augmented.
5 Experiments
We start the empirical validation of the proposed method by focusing on the pre-training of the mask generalor μ4 with surrogate masks created from procedurally generated graphs. Then, we introduce relevant baselines for our model and compare them on image generation tasks w.r.t. both synthetic and real-world data. We carried out our experiments with a GAN, even though the same principles can be extended to work with other methodologies. For this, noise is sampled at the input graph level, such that, V = =P]| Z) with Z Nx2_ N (0, 1). The code used to run the experiments is provided in the supplementary materials.
6

---

Published in Transactions on Machine Learning Research (07 (07/2
(b)
Figure 3: Sample of generated masks. For each one, the large figure is the aggregated mask, while the small ones are those associated with each node. The blue dots highlight the position of the accounted node. (a) is random graph like those used for pre-training; (b) and (c) are simple handcrafted ones.
(b) Nuu Rondon om
(8) Pautoom
Figure 4: Masks generated by pre-training random graphs (a) and on the Humans task's ones (b). For each group, the first two columns are samples from the Humans task, last two are random. In (b), performance clearly degrades out of distribution.
5.1 Surrogate Pre- Training
For surrogate pre-training, generate random graphs using the Barbasi-Albert (Albert & Barabási 2002) or a Redos-Renyi (ERDdS & R&wwi 1959: Gilbert 1959 model, with a number of nodes in the interval 30). The position of each node is determined by using Kamada-Kawai"s force-dirceted graph drawing algorithm (Kamada & Kawai 1989). These choices aim at generating (almost planar) graphs that can easily drawn on the plane, i.e. with only a few intersections among edges. More detailed information on the setup of the generators are provided in the supplementary matcrials. Target masks are computed as Equation (9) and the model is fitted minimize the binary cross entropy (BCE) loss. Figure shows sample results of the mask generation process. Fach one of the larger images corresponds to the full mask generated from the input graph (shown in blue and red), while the smaller images show masks relative to each node (in blue). The model produces masks that match the topology of the graph, even for complex structures. More interestingly, results show that the proposed architecture is indeed capable of generating localized node-level masks that capture the structure of the neighborhood samples (b) and (d), which contain handcrafted graphs, we see that this properly is preserved for graphs with a small diameter; even though, nodes with high connectivity tend to produce richer masks that may span it completely. Indeed, locality is preserved against over-smoothing which is typical of isotropic graph convolutional filters; conversely, the learned node-level masks diverse and properly localized. Figure [ additionally shows how pre-training randomly generated graphs can act as regularization by yielding a model able to perform properly for graphs outside the distribution of the downstream task.
each node. By looking at
5.2 Baselines
Comparing existing models to ours is not straightforward. As detailed in Section these methods lack flexibility concerning the objects they can generate and the representation of structural and semantic con- straints. Moreover, the tasks they address fundamentally differ from ours, as they rely on different inputs, and thus they cannot be directly compared to our approach. As described in Section 3 GraPhOSE) learns map the graph into layout mask by learning both node representations and node masks. We thus compare it to two baselines: both Conditioner) uses GNN-based encoder to learn node representations, while the second (FNN Conditioner) uses a feodforward ncural network (FNN) based one (i.c., it docs not use relational information). Considering Equation we can contextually schematize the baselines as computing mask M as
fixed (non-learnable) mapping to generate the node masks; the first. (GNN
M = bl (S (C)+
Σ SE (C))
(10)
JEN())

---

Published in Transactions on Machine Learning Research (07/ (224)
and, for the FNN-based one only, computing F as
F V= (V)
(11
where ()<ev()) indicates concatenation along a new axis, while, for ease of notation, SN (C) and SE, (C) denote operations applied over all the pixel coordinates w.r.t. nodes and edges, respectively. Note that the FNN Conditioner has still access structural information through the fixed masks which makes the comparison against GraPhOSE meaningful. For all the baselines (including GraPhOSEE), we use the same downstream model based on a simple stack of layers with 2D Convolutions, batch normalization (Toffe) & Szegedy 2015) and ReLU activation and residual connections (He et al.) 2016). Moreover, similarly to Johnson et al. (0188), cach layer may take as input both the previous layers's 's output and the downsampled layout mask to condition the generative process at the different processing steps. The discriminator is based on similar building blocks and receives the graph as input, together with the real/fake image. See Appendix | for more details on the implementation for each baseline. For GraPhOSE we pre-train the mask generator as described in Section 5.1 and then train the whole model on the downstream task. Note that, during the end-to-end training, we drop the reconstruction loss originally used during the pre-training. This is to allow for adapting the masking mechanism to the downstream task, removing any bias coming from the surrogate loss. All the basclines were trained under the same settings; more details can be found in Appendix A.2
5.3 Datasets
As previously mentioned we consider benchmarks based on synthetic and real-world data. We explicitly design the synthetic dataset, named PRO, to highlight the benefits of object-centric relational representa- tions. PRO consists of images containing simple stylized objects that can be rendered starting from a pose representation in which each keypoint has class and style attributes, i.e., from relational description of the image. Different objects are represented at different degrees of abstraction w.r.t. their appearance to make the task of generating objects, with a coherent style, from the representation more challenging. The style and visual appearance of the different rendered components vary smoothly with respect to the structure of each object (as it is often the case in the real world), making the adoption of appropriate inductive biases particularly appealing. For our experiments, we generated 100000 samples. For what concerns the second task, referred to as Humans, it consists of the problem of generating images of humans from key-points with class attributes (e.g., ankle, shoulder). To build a dataset of examples, we leveraged the existing MPII Human Pose (Andriluka et al., 2014), Market 1501 (Zheng et al. 2015 and DeepFashion (Liu et al. 2016) datasets. For the DeepFashion dataset, we use just the Fashion Landmark Detection and the In-Shop Re-
trieval subsets. The keypoint annotations for DeepFashion In-Shop Retrieval and Market 1501 are based on Zhu et al.] (2019), and were generated by using the open-source software OpenPose (Cao et al.) 2019
For DeepFashion Fashion Landmark we instead rely on MediaPipe's BlazePose (Bazarevsky et al. 2020) while MPI Human Pose already contains pose features. All annotations are reduced to the COCO keypoint standard (Lin et al. 2014) and used to generate the corresponding graphs with node coordinates normalized between 0 and 1, resulting in a dataset of roughly 300000 samples. Images from the real-world datascts have been downscaled (or padded) to a 64 x 64 shape. Images in PRO are generated with size 128 x 128 to ensure that objects are not too small. Further details on the datascts are provided in Appendix A.5 and A.6
6 Results
In the following section, we discuss the results obtained on the synthetic and real-world datasets. On the former, used small downstream model with around 28K parameters, while, the latter, more complex one, with roughly 5M parameters, was employed; details are provided in the appendix. Morcover, for the synthelic dataset, models that leverage relational representations receive inputs with 10% of the semantic node attributes randomly masked in 50% of the samples; we do so to show that structures can be used to generate coherent images from partial conditioning. Table [ shows the Frechet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al. 2016) for our model and the basclines w.r.t. both datasets, together with the Structural Similarity Index Measure (SSIM) (Wang et al.) 2004) for
8

---

Published in Transactions on Machine Learning Research (07 (07/2024)
(A) Enpor Craph
(FOOPPOOE leiNoPretoin IIJONN Coad. 1elFNV Coal (b)Con) Five (allnpet Corah
th Bander
(e) PNN Greal
(C) GRN Cood.
If CaPPeeNE (6) Sa SaPhutaai
(a) PRO
(b) Humans
Figure 5: Sample results for PRO (left) and Humans (right) tasks. Row (a) is the input graph. Generated samples come from: the PRO exact renderer (b left), the unconditioned downstream model (b - right), the FNN-based baseline (c), the GNN-based one (d), GraPhOSE without pre-training (e), GraPhOSE (f).
Table 1: Frechet Inception Distance (FID) (lower is better) and Inception Score (IS) (higher is better) for PRO and Humans datasets. Also the Structural Similarity Index Measure (SSIM) is reported for the PRO dataset.
PRO
Humans
MODELS
FID 4
IS t
SSIM
FID +
IS t
FNN Conditioner
103.86
3.2310.03
0.74
58.91
3.52:10.10
GNN Conditioner
105.2349.72
3.24
0.78
118.18478.54
3.58+0.16
GraPhOSE No-Pretrain
90.916.71
3.1110.02
0.7710.01
125.22184.14
1..2
GraPhOSE
66. 64.822
2.9240.01
0.84+0.01
54.46=7.57
3.41+0.12
the PRO dataset. GraPhOSE achieves a significantly better FID and SSIM score compared to the baselines. Indeed, in the PRO dataset, GraPhOSE outperforms the baseline both with and without pre-training for the FID measure. Note that SSTM measures the accuracy of image reconstruction, making a meaningful metric only for the PRO dataset, where the input graph fully describes the target image. For the human pose dataset, pre-training acts regularization allowing for reducing the variance among runs. As expected, the IS, which evaluates the distinctiveness and diversity of the generated samples, does not highlight significant differences among the models, as these propertics are not highly influenced by their architectural differences. We can then assess compliance w.r.t. the conditioning visually, in a qualitative manner.
In this regard, Figures 5a and 5b show results generated, by the compared models, for the two tasks. We can see that, in PRO, GraPhOSE, row (f), is clearly superior the other basclines, in particular when the surrogate mask further deviates from the object's visual appearance. The generation quality is consistent across different numbers of objects and scales, showing the approach is suitable to handle wide range of

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
lmpar Craph
(hi Peetnir
+
Io CraPh OSIB
K 7
X
Figure 6: Masks generated by GraPhOSE after pre-training (b) and after subsequent training the down-
stream task (c). (d), shows those generated by skipping pre-training, while (a) is the input graph
[a] Hand
Polygaa
Figure 7: Samples of individual node masks generated from GraPhOSE. The first column is the cumulative mask for the whole imput graph (superimposed with nodes in blue and edges in red). Individual node masks have the reference node highlighted in blue. Hand masks (a) are only a sub-sample for ease of visualization.
possible scenes. Moreover, the FNN Conditioner is not capable of handling missing attributes nodes, which hinders its flexibility, in particular for real-world settings. For Humans, again see that images generated by GraPhOSE have better visual appeal and compliance with the input pose, even when not all the key- points are present in the conditioning graph. Furthermore, notice how images from the FNN Conditioner, row (c), are visually worse than those from models with higher (worse) FID score. In general, relational representation plays significant role in guiding the generation toward coherent result, structure alone, i.e., even without semantics, can be leveraged characterize silhouettes, and neighboring information can help reconstruct missing features. Examples of this behavior are shown in Appendix B.1 Note that the generative models used in this work are purposely simple, as experiments focus the conditioning capabilities rather than image quality (i.e., detail, realism). The resulting image quality is comparable to that of architectures that use similar generators (Johnson et al.) 2018 Yan et al. 2016). Images of better quality can be obtained at the expense of increased computational and sample complexity by adopting more complex generators trained higher resolutions.
6.1 Task specific mask adaptation
The purpose pre-training on surrogate masks of random pose graphs is to learn a general mapping between the graph and the image representation of the structure it entails. However, this learned mapping might not produce the exact visual properties desirable for a specific downstream task. We wish adapt these properties while learning the target task, during the end-to-end training. Figure shows how this indeed
10

---

Published in Transactions on Machine Learning Research (07 (07/2024) /024)
Inpet Czaph
Figure 8: Results (b) of changing input nodes' position (a) while maintaining other inputs constant.
Figure Results of changing the input nodes' semantic attributes (i.e. color), for samples the PRO datasct. The input graph, with according colored nodes, is superimposed onto the generated image. Nodes belonging to the same finger change color in group otherwise the input graph would be out of distribution w.r.t. training data.
happens for the masks being generated before (b) and after (c) training on a target task. This adaptability is particularly relevant, as it shows that after pre-training the mask generator towards the correct direction, we can have learn the specifics of the target downstream task. Differently, the images generated by training GraPhOSE without pre-training on surrogate masks (d) do not visually resemble the input pose. These results suggest, the lack of structure in the mask leads to degradation of the performance, in particular when the target data distribution is not heavily biased towards a few poses (e.g., Humans contains mostly poses of people standing). Note that, even without pre-training, the model still leverages the relational inductive biases encoded by the input graphs and accounted for by our model architecture. This shows the usefulness of pre-training as a soft. regularization, which localizes node features w.r.t. corresponding portions of the image. Further examples of this property are shown the appendix. Figure shows how individual node masks preserve property of locality after training on the downstream task, even though no particular constraint, aside from architectural biases, enforces this. These results really meaningful, as localized masks allow, by GraPhOSE's design, for better local conditioning entails the learning of distributed representation, where each node describes specific part, rather than a collapsed representation in which all nodes contain a representation of the whole object. Further examples of this node-mask locality can be found in Appendix B.2
node level semantics. Moreover, it
6.2 Structure and style sensitivity analysis
To assess whether the model is able to disentangle structure and style, we experiment with providing it a series of human graphs with slightly different poses while fixing all the remaining attributes of the conditioning. As shown Figure generated figures are rearranged according to the changes in the input pose while the style mostly unchanged. Note that, however, in case of more significant changes the pose, the resulting images can be visually different. This emergent property nonetheless interesting and provides ground for future research. Similarly, assess the sensitivity of the method to incremental perturbations of the semantic node attributes for samples of the PRO dataset. Example results shown in Figure 9 We can see that the generated object's 's structure is not affected by the semantic changes. Only the colors change according to the changes in node attributes. This further highlights the disentanglement between structural and style control over the generated images.
7 Conclusions and Future Work
We propose GraPhOSE, a method to exploit attributed graphs as abject-centric relational inductive biases to flexibly condition image generation. Our approach has several advantages compared to standard approaches which result in scalable and flexible framework. Notably, we shone light on the properties of relational representation in this context, by showing how they be used to regularize and manipulate the generative
11

---

Published in Transactions on Machine Learning Research (07/2024)
model. We also provided novel synthetic dataset that pairs images with their object-centric relational rep- resentation, introducing benchmark to support further studies. Future research might target approaches to generate meaningful and coherent masks without relying on pre-training, or investigate methods to disen- tangle the diffcrent elements that contribute to the final generated image. We argue that our study has the potential of sparking new interest. object-centric relational represenlations for generative modeling and that the results presented here constitute the first building block for future rescarch this direction.
Acknowledgments
The authors wish to thank Daniele Zambon for the helpful suggestions provided during the development of this work.
References
Réka. Albert and Albert-László Barabási. Statistical mechanics of complex networks. Reviews of modern physics, 74(1):47, 2002.
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In TEEE Conference on Computer Vision and Pattern Recogration (CVPR), June 2014.
Davide Bacciu, Federico Errica, Alessio Micheli, and Marco Podda. A gentle introduction to deep learning for graphs. Newral Networks, 129:203-22 2201,2221121100 2020.
Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grund- mann. Blazepose: On-device real-time body pose tracking, 2020. URL. https: wwexxxiv.rrr///oo aa//abbs/0000
10204
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.ww wandb. com/ ware available from wandb.com.
Soft-
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://openrevies:
net/fforumniddBixsss]0333
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint ar arXim:2104 134788 2021.
Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Robert L Cook. Stochastic sampling in computer graphics. ACM Transactions on Graphics (TOG), 5(1): 51-72, 1986.
P ERDdS and A R&wi.. On random graphs i. Publ. math. debrecen, 6(290-297):18, 1959.
William Falcon and The PyTorch Lightning team. Pytorch lightning, 3 2019. URL https://www. pytorchlightning. ai
Edgar N Gilbert. Random graphs. The Annals of Mathematioal Statistics, 30(4):1141 1144, 1959.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine leorning, pp. 1263-1272. PMLR, 2017.
12

---

Published in Transactions on Machine Learning Research (07/2024)
lan J Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. Tn NTPS, 2014.
Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review on generative adversarial networks: Algorithms, theory, and applications. IEEE transactions on knowledge and data engineering, 35(4):3313-333 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 178, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In T. Guyon, U. Von Luxburg, S. Bengio, Н. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL huttss://proceeddngg pdf
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Newral Information Processing Systems, 43:6840-685/. 2020.
Yusuke Horiuchi, Edgar Simo-Serra, Satoshi Tizuka, and Hiroshi Ishikawa. Differentiable vendering-based pose-conditioned human image generation. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 3921-3925, 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125 1134, 2017.
Maor Ivgi, Yaniv Benny, Avichai Ben- Jonathan Berant, and Lior Wolf. Scene graph to image gen- eration with contextualized object layout refinement. In 2021 IEFE International Conference on Image Processing (ICIP), pp. 2428 2432. IEEE, 2021.
Yuming Jiang, Shuai Yang, Haonan Qju, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics (TOG), 414):11-1, 2022.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1219-1228, 2018.
Tomíhisa Kamada and Satorn Kawai. An algorithm for drawing general undirected graphs. Information Processing Letters, 31()) 15, April 1989. doi: 1010016/0020001196690000222 URL btttps://di... org/ 10. 1016/0000-0099(88)) 90102-6
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Represenlations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. HiRLhhttp://axxii org/ abs//412...1////////////a//aa1/a 1411.69800).
Diederik P. Kingma and Max Welling. Auto-cncoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http://arxiv..rrt wtwpr/iiiv.rrg////////11
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. DeepGCNs: Can GCNs go as deep as CNNs? In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9267-9276 2019.
Tsung-Y. Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1
13

---

Published in Transactions on Machine Learning Research (07/2024)
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances in neural information processing systems, 30, 2017.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Represenlations, 2017. URL htttps://openrevier wttp://rw.....ttoooooooeeessosa.
Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. Advances in neural information processing systems, 30, 2017.
Yifang Men, Yiming Mao, Yuning Jiang, Wei- Ying Ma, and Zhouhui Lian. Controllable person image synthesis with attributeddcoompsed gan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5084 5093, 2020.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.178, 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gener- ative adversarial networks. arXiv preprint 153::180200955 2018
Natalia Neverova, Riza Alp Guler, and lasonas Kokkinos. Dense pose transfer. In Proceedings of the European conference on computer vision (ECCV), pp. 123-138, 2018.
Taesung Park, Jun- Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang, Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems, 33:7198 7211, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeminng Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Deep learning on point sets for 3d
Xuelin Qian, Yanwei Fu, Tao Xiang, Wenxuan Wang, Jie Qiu, Yang Wu, Yu-Gang Jiang, and Xiangyang Xue. Pose-normalized image generation for person re-identification. In Proceedings of the European conference on compuler vision (ECCV), pp. 650 667, 2018.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arxiv:2204.00 01144.001255 25, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10684 10695, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe. Deformable gans for pose- based human image generation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3408-3416, 2018.
Jascha Sohk-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised lcarning using nonequiibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265, PMIRR, 2015.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Aduances in neural information processing systems, 32, 2019.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:19667-19679, 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error
visibility to structural similarity. IEEE transactions on image processing, 13(4):600 612, 2004.
Omry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019. URL https://githur: com/faacbboorreeerrr///yrrr
Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Attribute2image: Conditional image generation from visual attributes. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-14, 2016, Proceedings, Part IV 14, pp. 776-791. Springer, 2016.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re- identification: A benchmark. In Computer Vision, IEEE International Conference on, 2015.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofci Wang, and Xiang Bai. Progressive pose attention transfer for person image generation. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 2347-2355 2019.
15

---

Published in Transactions on Machine Learning Research (07 (07/2024)
A Implementation Details
In the following we clarify some details on the settings used train the models, on the specifics of some models' architectures, as well further details surrogate masks and the hardware and software we used to run the experiments.
A.1 Hardware and Software
The code to reproduce the experiments and generate the PRO dataset is available online It is all written in Python 3.9, leveraging Pytorch (Paszke et al. 2019 and Pytorch Lightning (Falcon & team 2019 to define models and data, while Hydra Yadan 2019) is used to manage the experiment configuration. Weights & Biases (Biewald 2020) is used to log and compare experiment results. We run all the experiments on an NVIDIA RTX A5000 GPU equipped with 24GBs VRAM.
A.2 Training settings
Each model trained under the same settings and metrics are computed over three different runs each. In particular, we use the Adom (Kingma & Ba 2015 optimizer, with cosine annealing (Loshchilov & Hutter 2017) learning rate schedule with period 300 for the PRO datasct and 150 for the Humans onc, starting learning rate 0.002 and final learning rate 0.00002. Models are trained up to 300 (PRO) or 150 (Humans) epochs each, with a batch size of 64 and early stopping with patience 50 on the FID. Notice that the learning rate for the parameters of the pre-trained mask generator used in GraPhOSE was reduced to & (PRO) or 100 (Humans) with respect to the learning rate of the other parameters, in order to avoid catastrophic foregetting at the beginning of training. This procedure could be enhanced by slowly raising the learning rate of the pre-trained parameters back to the value of other parameters, as training epochs go by, with the aim of favoring mask adaptation aftor the training has stabilized during the first epochs. The reduction was much stronger for Humans in order to counter balance the heavy bias the dataset had towards poses of people standing up with arms straight down. However, this reduction factor is indeed hyper-paramcter that can be effectively used to allow for more or less deviation from the surrogate masks, depending on the downstream task characteristics.
1
A.3 Surrogate Mask
Referring to Equation
we explicitly define T.j as
dj = IP: - P, 12,12
4
aij = arctan 2 (p, - -p))
di
10,15 = dy cos (axz)) 2
sin (azy)² )
a²
+ dy
(12)
to.i = di- sin (arr)
2
cos (a:;)² ?
a²
di
tc.ij
dij a²
sin (aiz) cos (aiz)
tc
Tij = bojs
16,55
where vectors p denote node coordinates, arctan 2 is the element-wise 2-argument arclangent and denotes parameter that regulates the scaling ratio of the second dimension w.r.t. the first one. In our experiments is set to 10.
16

---

Published in Transactions on Machine Learning Research (07/202) /024)
Figure 10: Examples of objects present in the PRO datasct. From left to right: Pie, Scissors, Hand, Robotic
Arm, Hollow Polygon, Filled Polygon, Lattice
A.4 Random Graphs for Pre-Training Training
Referring to Section 5.1 we set the Barbasi-Alber model's number of edges for new nodes parameter to 1 in 90% of the cases and to 2 in the remaining 10%, if the number of nodes less than 10. For graphs with more than 10 nodes, is always set Erdos- Renyi model's odge probability is set to exp (2) 0.95, where is the number of nodes.
A.5 PRO Dataset
Our synthetic Rese-eepreeeetable Objects (PRO) dataset consists of square images (i.c., with same height and width) rendered from random graphs that represent certain object's parts and their color. A graph representing an object has a fixed connectivity (i.e., adjacency matrix) and each node has color (from a pool of possible ones) and class attributes together with a coordinate in 2D space. Coordinates and colors, for each node in a sample, are generated randomly within reasonable boundaries (e.g. to stay within the image to comply to specifie structure, like that of regular polygon). Figure 10 some examples of each object type in our dataset; the graph that describes the object has the edges superimposed in white, while the nodes are colored accordingly to their attributes. Images from the PRO dataset can contain multiple objects, however we experimented with up to four objects for image, in order to keep their size reasonably big. Regarding the objects' structural constraints, we highlight the following properties: Pie has an inner angle of up to 135%, as the two pie tips have the same class and would not be possible to discriminate which part of the pie has to be filled otherwise. Pie nodes have always the same color. Scissors have the structural constraint that the angle between the blades and the handles must be the same and be contained between 30" and 90" Also blades and handles must respectively have the same color. For the Hand object we have constraints on the maximum relative angle between each phalanx, in order to avoid impossible orientations; moreover, fingers must have consistent color. Hands can uniformly be either 'right' or "left" hands. The Robotic Arm, similar to the Hand, has constraints on the relative angles between its parts, to similarly avoid collapsed configurations. Additionally, the number of segments their color must not change. Similarly, also the prongs must have a consistent coloring. Hollow Polygon and Filled Polygon must always have all nodes of the same color and the presence of a 'center' node determines if the polygon is filled or not. We experiment with polygons up to 8 vertices. The Lattice is the only structure with randomized connectivity. Disk distribution (Cook, 1986) and are connected if their distance is less than 20% of the whole image width. The lattice is also the only object in which each node has a random color and the segment connecting two nodes is colored with a linear gradient going from one node's color to the other's.
the arm can vary between and 7 and
nodes, uniformly dist.ibbttdd between 3 and 9, are sampled from Poisson
Table contains a summary of the classes for cach object's node. We would like to underline that the presence of structure in the assignment of colors and classes lo object nodes is not just. to make them visually consistent, but is fundamental in order make it possible to reconstruct missing attributes by using relational information and acquired knowledge on how said structure behaves.
Regarding dataset size, in our experiments we generated 100000 images. We employed 1000 of these as validation set, and another 1000 as test set. Images were sampled so that the dataset contains a uniform distribution of different objects. Morcover, also the number of objects that appear an image at once, between 1 and 4, is uniformly distributed across the dataset. Train, validation and test splits, while random, preserved these uniform distributions.
17

---

Published in Transactions on Machine Learning Research (07/2024)
Table 2: Node classes for each object.
OBJECT
CLASSES
Pic
center, tip pivol, blade, handle wrist, finger base, arm, prong vertex vertex, peneter I vertex
Scissors
Hand
Robotic Arm
Hollow Polygon
Filled Polygon
Lattice
Table 3: The Encoder d
LAYER
H-PARAMS
INPUT
OUTPUT
MultiEmbeddding
8, cal
X
PoseConv
8,8,8
O P, A
0
BatchNorm
0 0
0
ReLU
0
PoscConv
8, 16, 16
O, O.P.A A
0
BarchNorm
0
0
ReLU
PoseConv
16,32,
O, O.P.A
Sigmoid
0
0
A.6 Humans Dataset
The Humans dataset consists of roughly 300000 images, coming from MP11 Human Pose (Andriluka et al, 2014), Deep Fashion (Liu et al. 2016) and Market 1501 (Zheng et al. 2015 which are all widely adopted benchmarks for pose estimation and other human body related tasks. Out of these, around 3000 images, were used for validation and an analogous size for testing. Train, validation, and test sets were partitioned random but the proportion of samples from the original benchmarks was maintained. Within the Humans task, the only object type is Person, while there are 14 node classes (left and right shoulders, elbows, wrists, ankles, knees, hips plus base of the neck and top of the head). Note that, differently from the PRO dataset, these real-world images contain different backgrounds and lighting conditions. We do not address the explicit conditioning over these properties in our model, however these could be incorporated via a vector representation that acts as a global graph attribute.
A.7 Architectures
In the following we give more precise specification for the architecture we used in our experiments, while specifying that these represent only a possible way of implementing our approach and deeper or more complex architectures might be better suited for different tasks. Coherently with previous notation, Tables 3 to describe the architectures of the different components in our framework; while Tables g and for clarity, describe recurring building blocks (i.e., used multiple times) inspired by Brock al. (2019 In particular, clarify the meaning of some layers: MultiEmbedding simply denotes multiple embedding layers (Bengio ett al. 2000) with the same output size, used to obtain continuous representation for different discrete node attributes. The first hyperparameter refers to the embedding size, while the second describes the way in which embeddings of different. features are combined inlo a single embedding. In particular, cal refers to concatenation along the feature dimension. The usage of embedding layers can be exchanged for multiple lincar layers, if the input features are already continuous. PoseCone described in Equation 3) by respectively naming its three hyperparameters h and o, we can further specify that, in this inplementation, Is and gi linear layers with input size 2 is the size of pt)
layer that implements the function
18

---

Published in Transactions on Machine Learning Research (07/2024)
Table 4: The FNN-based Encoder Ve.
LAVER
H-PARAMS
INPUT
OUTPUT
MultiEmbedding
8, cat
X
0
FNNPoseConv
8,8
O P, A O 0
0
BarchNorm
0
ReLU 8,16 BatchNorm
0
ENNPoseConv
O, O.P.4
0
ReLU
0
FNNPoseConv
16,32
P,P,A
0
Sigmoid
0
0
Table 5: The Mask Generator pe-
LAYER
H-PARAMS
INPUT
OUTPUT
PoseConv
3,8,8
P Z, P. A O O O,P, P.PAA
BatchNorm
0
ReLU
0
PoseConv
8,32, 32
BatchNorm
O
ReLU
0
0
PoseConv
32, 128, 128
O,P, A
O
Reshape
128, [8, 4,
O
O
2oseConv2D
8, 16
O.
0
ConyBlock2D
12,2
0
0
PoseConv2D
32, 16
0,
PoseConv2]
16,8
O.
BatchNorm2D
0
ReLU
0
0
Conv2D
8,
0
0
Sigmoid
O
O
and output size h, while yy is linear layer with input size h and output size Analogously, PoseConv2D
refers to a layer implementing Equation (5), where is a ConeBlock2D with in = 2-1, out = and = 2, followed by (H) = (H) @ H. Is is a times two upsampling followed by a 1 x 1 2D convolution with 0 padding, input size and output size o and, finally, 9g is also a 2D convolution with padding and both input and output size set to o. In this case i the first hyperparameter of PoseConv2D, while the second. FNNPoseCone the FNN equivalent of PoseConv, computing
h (h:) + Inr (hallp:),
(13)
where, naming the first hyperparameter and the second o, far and 9s are both linear layers, the former with input size i i+2 and output size o and the latter with input size and output size o. Note that, when using the FNN Conditioner, the PoseConv layers of the discriminator ones. Furthermore, note that, unless differently stated, all Conv2D operations use kernel of size 3 3 with stride and padding set to 1. All layer weights are normalized through spectral normalization (Miyato et al. 2018), Upsample operations are implemented through bilinear interpolation, while Downsample ones use 2D average pooling. For completeness, we clarify that the Sum(H, W) operation in Table refers to summing the tensor Og E BBBxXxXXX over the last two dimensions, thus obtaining the matrix O₂ E BBC
swapped for equivalent FNNPoseConv
Tables 10 and show, respectively, the architectures of the downstream generator and discriminator, used for the human dataset. Tn such case the GraPhOSE architecture remained the same, although the
19

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Table 6: The Downstream Generator.
LAYER
H-PARAMS
INPUT
OUTPUT
ConyBlock2D
32,32,
L
ConvBlock21
32,32,
0
DonvBlock2D
32, 16,
0
ConvBlock2D
16, 16, 1
0
ConvBlock2D
16,8,
0
BatchNorm2D
0
ReLU
0
Conv2D
8,3
0
0 0
Tanh
0
Table 7: The Downstream Discriminator.
LAYER
H-PARAMS
INPUT
OUTPUT
MultiEmbedding
8, cat
O
PoscConv
8,8,8
O P. A
O
BarchNorm
O
O
ReLU
O
O
PoseConv
8, 1616, 16
O,, P.A
O
BatchNorm
O
O
ReLU
O
O
PoseCony
16,32,
O P,A
O
DConv DConyBlock2D
3,8,2
02
DConv D
16,2
02
O2
DCony Block2D
16, 16,2
O2
O2
DCony Block2D
16, 12,
O2
O2
D
32, 32,2
02
O2
ReLU
02
02
Sum
H, W
02
0
Linear
64,32
03
ReLU
O3
03
Lincar
32, 1
O3
O3
number of hidden fcatures increased by factor of two and μa dropped the ConvBlock2D, the target images had shape 64 x 64, hence further upsampling was required. The main differences present at the discriminator and generator level, as for this more complex task we borrowed the building blocks of BigGAN (Brock et al. 2019). In particular, DConvBlock2DD* has slightly different architecture in which the first Conv2D layer outputs 2 out features and followed by a ReLU and another Conv2D layer that outputs out features. ConnBlock2DD instead, takes as additional input the downsampled layout mask L*, which goes through Conv2D layer that outputs out features; these are concatenatod the output of the Conv2D the standard input and go through a second Conv2D(2 oul, out) layer, before the skip connection. This serves the purpose of conditioning the generative process at different scales, as, in this case, the downstream generator starts from 1 x 4 x 4 feature map, Z, which is upsampled at each step, instcad of directly using the layout mask and always keeping the same feature map size. For completeness, Z represents additional latent conditioning, required to account for the background, and is obtained by reshaping the output of a linear layer with output size 16, which takes as input vector normally distributed noise of size
Table 12 summarizes the parameter counts for GraPhOSE and the baselines, differentiating between the smaller versions used for the PRO dataset and the larger ones used for Humans.
20

---

Published in Transactions on Machine Learning Research (07/2024)
Table 8: The ConvBlock2D (in, out, up).
LAYER
H-PARAMS
INPUT
OUTPUT
BatchNorm
H
O1
ReLU
0
O1
U Upsample
up up in, out
01
O1
U Upsample
H
02
Conv2D
01
01
Conv2D
in, oul.
O2
02
Sum
01, O2
0 02
Table 9:
The DConvBlock2D (in, out, down).
LAYER
H-PARAMS
INPUT
OUTPUT
ReLU
H
O
Conv2D
in, out down in, out down
O1
01
Downsample
O
01
Conv2D
H
0
Downsample Sum
O2
02
O1,0.
02
0
02
A.8 Metrics
We chose Frechet Inception Distance (Heusel et al. 2017) and Inception Score (Salimans et al. 2016) as metrics for our generative models as both are commonly used to asses generation quality. However, only FID is effectively useful in asscssing how much the generated samples resemble those coming from the real distribution, as this metric compares the similarity between distribution images coming from the generator and those coming from the dataset. Inception Score is limited evaluation of some properties of the generated images distribution, hence it is less useful as metric of comparison between models because it can only tell if the model is capable of generating images that are sufficiently distinct and recognizable, even though they might actually be completely different from those of the real distribution. In other words, IS is only influenced by the capability of the generative model of producing samples that can be distinctively allocated to a class and that, at the same time, are diverse within each class. In our setting, the differences between GraPhose Phose and each baseline are not expected impact such properties, which are mostly lied the downstream model, which always the same for each task. This makes the resulting Inception Score quite uninformative a way of comparing the various models; nonetheless, it highlights that learning the object's 's relational masks does not indeed harm the appeal of the generated samples. For the PRO dataset also reported the Structural Similarity Index Score (SSTM) (Wang et al. 2004). This metric is used of image reconstruction accuracy, hence it is suitable to evaluate generation quality only in situations where the input lo the model uniquely and completely describes the image to be generated. That is the case for the PRO task, as there a deterministic mapping between the graph and the resulting image. However, that is the case for the Humans dataset, in which the input graph just describes the person's pose but carries no information regarding the clothing the background, hence the reason for not employing such metric in that scenario.
measure
B Additional results
The following is a collection of additional results, comprising further examples of what already discussed in the main paper, and also images generated by training models with slight architectural changes.
21

---

Published in Transactions on Machine Learning Research (07/2024)
Table 10: The Downstream Generator used for the human dataset.
LAYER
H-PARAMS
INPUT
OUTPUT
ConvBlock2DD
1,512,
Z,L'
ConvBlock2D
512, 256,
O,L'
D
256, 128, 2
O,L'
D
128, 64,
O,L"
Attention
64
ConvBlockkDD*
64, 64,32,2
O,L'
BatchNNrrm21
0
0
ReLU
0
0
Conv2D
32,3
0
0
Tanh
0
0
Table 11: The Downstream Discriminator used for the human dat aset.
LAYER
H-PARAMS
INPUT
OUTPUT
MultiFabbedding
8, cal 16, 16
X
O
PoscConv
O P, A
O
BatchNorm
O
O
ReLU
O
O
PoseConv
16, 16,32,32
O1, P,A
O
Batch.Nomm
O
O
ReLU
O
O
PoseCony
32, 6,,4, 64
O2, P,A
O
DConvBlock2D
3,32,2
02
Atlention
32
02
02
DConvBlock2D*
32, 64, 2
02
02
DConvBlock2D
64, 128,2 2
02
02
DConvBlock2D
128, 256, 2
02
O2
DCony Block2D*
256, 512, 2
02
02
ReLU
0g
02
Sum
H, W
02
02
Linear
576, 512
01|
03
ReLU
O:
O3
Linear
512, 1
O₂
O3
B.1 Missing attributes
As discussed, our formulation allows for sub-conditioning the pose graph, i.e., having nodes without any semantic attribute assigned, in order to let the generative process decide how to treat them. Figure II shows some examples of this behavior, which white nodes represent nodes without any semantic attribute. We can see that the generation mostly consistent when the attributes can be recovered from neighbouring nodes. However, in cases in which the information needs to be reconstructed through multiple hops (e.g.. the scissors handle) the model uses the nearest color instead. This might be the result of biases in the data. Mind that we use the term reconstruction looscly here, as the model is trained with generative objective, hence we might expect different results we train it on a striet reconstruction lask. Figure 12 instead, shows what happens when we have a growing chain of missing attributes. We can see that semantics are coherently filled in, by GraPhOSE, up a certain number of subsoquent masked nodes. This is, part, due architectural constraints, the number of hops information can traverse depends on the number of message passing steps. On the other hand, the usage of graph convolution operators that try to prevent
22

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Table 12: GraPhOSE and baselines approximate parameter count, for the two tasks, divided between con-
ditioning, downstream generator and discriminator.
PRO
Humans
MODELS
Conditioning Generator Discriminator Conditioning Generator Discriminalor
FNN Conditioning
2K
27K
23K
6K
8.6M
5.2M
GNN Conditioning
5K
27K
26K
17K
8.6M
5.2M
GraPhOSE
63K
27K
26K
2.4M
8.6M
5.2M
Sample images generated from GraPhOSE presence of missing semantic attributes: from left to right, Hand, Robotic Arm, Filled Polygon and Scissors. White nodes have their semantic attributes masked, while others are colored according their color attribute.
Figure 11:
over-smoothing might also play role in this. However, notice we did not train model to specifically address high chances of partial conditioning, ie., missing attributes; different results be expected if we purposcfully train it on datasets with such characteristics. Even though, in such experiments, the color relationships between object components are rather simple, we think these results, tied with the architect ural constraints involved, can be a promising basis for further, and more specifically aimed, experimentation.
B.2 Mask adaptation
shows additional examples of how the pre-learned masks consistently adapt to the downstream task during the end-to-end training, while figure 14 shows how directly training on the downstream task leads masks that are not visually meaningful. This is particularly severe for the case of human poses, where the dataset is heavily biased towards few particularly common ones.
Figure 13
Figure 15 shows further examples of how node-mask locality is preserved after downstream training, and adapled to the specifics of the objects in the task. We can notice that the Pie (e) is the ouly object for which this locality seems to be lost, however it is not surprising as its graph representation diameter is only 2, with just 3 nodes, hence it is harder to avoid over-smoothing, For graphs with larger diameter and/o /or number of nodes, we see that locality is retained.
23

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Figure 12: Example of how GraPhOSE behaves, with graph with n nodes, in presence of missing altributes, the chain of subsequent ones increases in length from 0 (left) to (right). White nodes represent missing attributes.
(a) Empest Croph
Hiilore
Aftes
Figure 13: Random sample of humans masks generated by model after pre-training (b) and after subse- quent training on the downstream task (c). The top row (a) shows the corresponding input graph.
Figure 14: Samples of humans masks generated by GraPhOSE's mask generator after end-to-end training without surrogate mask pre-training. The input graph is superimposed in blue (nodes) and red (edges).
24

---

Published in Transactions on Machine Learning Research (07/202) /024)
(a) Hand
Polygon
(c) Filled Polygon
(d) Sciswors
Figure 15: Samples of individual node masks generated from GraPhOSE. The first column the cumulative mask for the whole input graph (superimposed with nodes in blue and edges in red). Individual node masks have the reference node highlighted in blue. Hand masks (a) are only sub-sample for ease of visualization.
25


Reviews:
[{"review_id": "n3QEWxJNfo", "reviewer": "Reviewer_GbKP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method looks interesting.\n2. The generated images can be flexibly controlled by graphs.\n\nWeaknesses:\n1. My major concern lies in the quality of the generated images. According to Figure 5, the generated image seems blurred. In fact, some relevant methods from some years ago [*1, *2] may already outperform this paper in terms of generation quality.\n2. Each image in the training set needs to be annotated with a graph, which not only increases the cost, but also hinders the proposed method from being applied to more general image generation tasks.\n\n[*1] CVPR'18, Image Generation from Scene Graphs.\n[*2] ECCV'16, Attribute2Image: Conditional Image Generation\nfrom Visual Attributes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel method for conditioning image generation using object-centric relational representations. The methodology involves conditioning the generation of objects in an image on an attributed graph that represents their structure and semantic information. The approach uses a neural network to generate a 2D, multi-channel layout mask of the objects, which serves as a soft inductive bias in the downstream generative task. The proposed method outperforms some baselines on a new PRO benchmark consisting of a synthetic dataset of images paired with their relational representation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses.", "broader_impact_concerns": "I have no concern about the broader impacts and the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ngSEJAtxsW", "reviewer": "Reviewer_bkj9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1)   This is the first work exploring conditional image generation using object-centric relational representations.\n2)   They introduce a new benchmark for image generation, comprising a synthetic dataset pairing images with their relational representations.\n\nCons:\n1)   The proposed method’s advantages in flexibility, generality, and effectiveness compared to existing methods remain uncertain due to the lack of comparative experiments with Scene Graph Conditioned Image Generation and Pose Conditioned Image Generation methods. Although the existing models have limited object diversity and often require diverse inputs, conducting experiments with consistent design is crucial for validating the claim.\n2)   Figure 8 demonstrates the consistency of generated images across different node positions. However, there is a lack of experimental validation regarding the variation of node semantic information and its impact on the generated results.\n3)  Could the author offer a more detailed description of the dataset, such as its size, and how the training and testing data are partitioned?\n4)  I suggest the authors following [1] to perform a more fair and more comprehensive evaluation. Specifically, more metrics on more datasets should be used, such as SSIM and user study etc.\n5)  No explanation about the meaning of A in Equation 2.\n\n[1] “Pose Guided Person Image Generation”, NeurIPS 2017.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new method for conditioning image generation using object-centric relational representations. By leveraging attributed graphs to represent structure and semantic information, it allows for effective conditioning of object generation in images. Implemented through a neural network with 2D and graph convolutional operators, empirical results demonstrate its effectiveness compared to baselines on a synthetic dataset.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness part for the required changes.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "djILYZO01O", "reviewer": "Reviewer_CsdN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nClear Methods and Unified Framework: The paper's strength lies in its clear exposition of methods and the proposal of a unified framework for conditioning image generation. This clarity facilitates understanding and implementation, enabling researchers to easily adopt and extend the proposed approach in their work.\n\nWeakness:\n\nLack of Clarity on Dataset Design and Generation: The paper could benefit from providing more detailed information on how the test datasets were designed and generated. Additionally, the lack of support from other open datasets may limit the demonstration of the method's efficacy across diverse domains and data distributions. Incorporating additional datasets or providing insights into dataset selection and generation processes could strengthen the paper's empirical validation and broaden its applicability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method for conditioning image generation using object-centric relational representations. The paper addresses the limitation of existing approaches by proposing a unified method for representing both structural and semantic conditioning at various granularity levels in image generation tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can author exposure more details of the Test Set? For example:\n\n1. The distribution of the dataset: The paper should provide information on the size of the test dataset, including the number of images and the distribution of objects and classes within these images. This information helps assess the diversity and representativeness of the dataset.\n\n2. Table 2 and Object Classes: If Table 2 lists all the objects and classes used in the experiment, it is essential to confirm whether this coverage is sufficient for constructing a comprehensive test set. An ideal test set should encompass a wide range of objects and classes to evaluate the generative model's ability to handle diverse scenarios.\n\n3. Generalizability: It's crucial to assess whether the test set generalizes well to measure the quality of the generative model accurately. The test set should cover a variety of object configurations, backgrounds, lighting conditions, and other factors to evaluate the model's robustness and generalization capabilities effectively.\n\nCan this paper leverage widely used benchmarks or tasks to enhance the validation and applicability of the proposed model?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has two contributions:

1) a methodological framework for conditioning images on object-centric relational representations.
2) a new synthetic benchmark dataset with images and conditioning representations.

The reviewers appreciated the contributions of the paper, but also had some reservations especially regarding image quality and comparison to prior work. The authors addressed these questions to some degree. In the final revised version the authors are encouraged to revise the paper reflecting these comments. This will make the paper stronger.
Decision: Accept (poster)