Module	Output	Implementation
find[qatt]	p	W1T([x; v]) + b1
filter[qatt](p)	p	p ⊙ (W1T([x; v]) + b1)
with-relation[qatt](p1, p2)	p	max(p2)p1 ⊙ MLP([x; v1; v2])
project[qatt](p)	p	max(p)find(qatt) ⊙ MLP([W2; v1; v2])
count(p)	N	number(∑(p), σ2)
exist(p)	B	greater-equal(p, 1)
greater-equal(a : N, b : N)	B	greater(a, b) + equal(a, b)
less-equal(a : N, b : N)	B	less(a, b) + equal(a, b)
equal(a : N, b : N)	B	∑k=0KPr[a = k] Pr[b = k]
less(a : N, b : N)	B	∑k=0KPr[a = k] Pr[b > k]
greater(a : N, b : N)	B	∑k=0KPr[a = k] Pr[b < k]
and(a : B, b : B)	B	a*b
or(a : B, b : B)	B	a+b-a*b
number(m : F, v : F)	N	Normal(mean = m, var = v)
sum(a : N, b : N)	N	number (a mean + b mean, a var + b var)
difference(a : N, b : N)	N	number (a mean - b mean, a var + b var)
division(a : N, b : N)	N	number(a mean + b mean, a^2b^3, b^2mean, a^2b^2mean + b^2b^2mean)
intersect(p1, p2)	p	p1 · p2
discard(p1, p2)	p	max(p1 - p2, 0)
in-left-image(p)	p	p s.t. probabilities for right image are 0
in-right-image(p)	p	p s.t. probabilities for left image are 0
in-at-least-one-image	B	macro (see caption)
in-each-image	B	macro (see caption)
in-one-other-image	B	macro (see caption)
Table 3: Implementations of modules for NLVR2 NMN. First five contain parameters, the rest are deterministic. The implementation of count shown here is the Sum-count version; please see Section 4 for a description of other count module varieties and a discussion of their differences. 'B' denotes the Boolean type, which is a probability value ([0..1]). 'N' denotes the Number type which is a probability distribution.  $K = 72$  is the maximum count value supported by our model. To obtain probabilities, we first convert each Normal random variable  $X$  to a categorical distribution over  $\{0,1,\dots,K\}$  by setting  $\operatorname*{Pr}[X = k] = \Phi (k + 0.5) - \Phi (k - 0.5)$  if  $k\in \{1,2,\dots,K - 1\}$ . We set  $\operatorname*{Pr}[X = 0] = \Phi (0.5)$  and  $\operatorname*{Pr}[X = K] = 1 - \Phi (K - 0.5)$ . Here  $\Phi (\cdot)$  denotes the cumulative distribution function of the Normal distribution.  $W_{1},W_{2}$  are weight vectors with shapes  $2h\times 1$  and  $h\times 1$ , respectively. Here  $h = 768$  is the size of LXMERT's representations.  $b_{1}$  is a scalar weight. MLP denotes a two-layer neural network with a GeLU activation (Hendrycks and Gimpel, 2016) between layers.  $x$  denotes a question representation, and  $v_{i}$  denotes encodings of objects in the image.  $x$  and  $v_{i}$  have shape  $h\times |\mathcal{B}|$ , where  $|\mathcal{B}|$  is the number of proposals.  $p$  denotes a vector of probabilities for each proposal and has shape  $1\times |\mathcal{B}|$ .  $\odot$  and  $[;\]$  represent elementwise multiplication and matrix concatenation, respectively. The expressions for the mean and variance in the division module are based on the approximations in Seltman (2018). The macros execute a given program on the two input images. in-at-least-one-image macro returns true iff the program returns true when executed on at least one of the images. in-each-image returns true iff the program returns true when executed on both of the images. in-one-other-image takes two programs and returns true iff one program returns true on left image and second program returns true on right image, or vice-versa.