RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Checklist
For all authors…

Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] Contributions are clearly stated in the third and fourth paragraphs of the introduction.

Did you describe the limitations of your work? [Yes] The proposed learning algorithms are designed for smooth and decomposable PCs.

Did you discuss any potential negative societal impacts of your work? [N/A]

Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]

If you are including theoretical results…

Did you state the full set of assumptions of all theoretical results? [Yes] All theorems in the main text formally state all assumptions.

Did you include complete proofs of all theoretical results? [Yes] Proofs are included in the appendix.

If you ran experiments…

Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Experiment setup is detailed in the main text and the appendix.

Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Training details are provided in the appendix.

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A]

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details about computational resources can be found in the appendix.

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

If your work uses existing assets, did you cite the creators? [Yes] We cited all PC learning algorithms as well as the adopted datasets.

Did you mention the license of the assets? [Yes] We specified that the used algorithms are publicly available.

Did you include any new assets either in the supplemental material or as a URL? [Yes] We included our code in the supplementary material.

Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]

If you used crowdsourcing or conducted research with human subjects…

Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Hardware specifications
All experiments are performed on a server with 32 CPUs, 126G Memory, and NVIDIA RTX A5000 GPUs with 26G Memory. In all experiments, we only use a single GPU on the server.
C.1 Datasets
For MNIST-family datasets, we split 5% of training set as validation set for early stopping. For Penn Tree Bank dataset, we follow the setting in Mikolov et al. [ 29 ] to split a training, validation, and test set. Table 3 lists the all the dataset statistics.
HCLT structures.
Adopting hidden chow liu tree (HCLT) PC architecture as in Liu and Van den Broeck [ 25 ] , we reimplement the learning process to speed it up and use a different training pipeline and hyper-parameters tuning.
EM parameter learning
We adopt the EM parameter learning algorithm introduced in Choi et al. [ 4 ] , which computes the EM update target parameters using circuit flows. We use a stochastic mini-batches EM algorithm. Denoting \theta^{\mathtt{new}} as the EM update target computed from a mini-batch of samples, and we update the targeting parameter with a learning rate \alpha : \theta^{t+1}\leftarrow\alpha\theta^{\mathtt{new}}+(1-\alpha)\theta^{t} . \alpha is piecewise-linearly annealed from [1.0,0.1] , [0.1,0.01] , [0.01,0.001] , and each piece is trained T epochs.
Hyper-parameters searching.
For all the experiments, the hyper-parameters are searched from

h\in\{8,16,32,64,128,256\} , the hidden size of HCLT structures;

\gamma\in\{0.0001,0.001,0.01,0.1,1.0\} , Laplace smoothing factor;

B\in\{128,256,512,1024\} , batch-size in mini-batches EM algorithm;

\alpha piecewise-linearly annealed from [1.0,0.1] , [0.1,0.01] , [0.01,0.001] , where each piece is called one mini-batch EM phase. Usually the algorithm will start to overfit as validation set and stop at the third phase;

T=100 , number of epochs for each mini-batch EM phase.

The PC size is quadratically growing with hidden size h , thus it is inefficient to do a grid search among the entire hyper-parameters space. What we do is to fist do a grid search when h=8 or h=16 to find the best Laplace smoothing factor \gamma and batch-size B for each dataset, and then fix \gamma and B to train a PC with larger hidden size h\in\{32,64,128,256\} . The best tuned B is in \{256,512\} , which is different for different hidden size h , and the best tuned \gamma is 0.01 .
Sparse PC (ours).
Given an HCLT learned in Section C.2 as initial PC, we use the structure learning process proposed in Section 5 . Specifically, starts from initial HCLT, for each iteration, we (1) prune 75% of the PC parameters, and (2) grow PC size with Gaussian variance \epsilon , (3) finetuing PC using mini-batches EM parameter learning with learning rate \alpha . We prune and grow PC iteratively until the validation set likelihood is overfitted . The hyper-parameters are searched from

\epsilon\in\{0.1,0.3,0.5\} , Gaussian variance in growing operation;

\alpha , piecewise-linearly annealed from [0.1,0.01] , [0.01,0.001] ;

T=50 , number of epochs for each mini-batch EM phase;

for \gamma and B , we use the tuned best number from Section C.2 .
HCLT.
The HLCT experiments in Table 1 are performed following the original paper (Code https://github.com/UCLA-StarAI/Tractable-PC-Regularization ), which is different from the leaning pipeline we use as our inital PC (Section C.2 ).
SPN.
We reimplement the SPN architecture ourselves following Peharz et al. [ 34 ] and train it with the same mini-batch pipeline as HCLT.
IDF.
We run all experiments with the code in the GitHub repo provided by the authors. We adopt an IDF model with the following hyperparameters: 8 flow layers per level; 2 levels; densenets with depth 6 and 512 channels; base learning rate 0.001; learning rate decay 0.999. The algorithm adopts an CPU-based entropy coder rANS.
BitSwap.
We train all models using the following author-provided script: https://github.com/fhkingma/bitswap/blob/master/model/mnist_train .
BB-ANS.
All experiments are performed using the following official code https://github.com/bits-back/bits-back .
McBits.
All experiments are performed using the following official code https://github.com/ryoungj/mcbits .
C.4 Details of Section 6.2
For all experiments in Section 6.2 , we use the best tuned \gamma and B from Section C.2 and hidden size h ranging from \{16,32,64,128\} . For experiments “What is the Smallest PC for the Same Likelihood?”, the hyper-parameters are searched from

k\in\{0.05,0.1,0.3\} , percentage of parameters to prune each iteration;

\alpha , piecewise-linearly annealed from [0.3,0.1] , [0.1,0.01] , [0.01,0.001] ;

T=50 , number of epochs for each mini-batch EM phase;

For experiments “What is the Best PC Given the Same Size?”, we use the same setting as in Section C.3 .
Figure 1 : Histogram of parameter values for a state-of-the-art PC with 2.18M parameters on MNIST. 95% of the parameters have close-to-zero values.; Histogram of parameter values for a state-of-the-art PC with 2.18M parameters on MNIST. 95% of the parameters have close-to-zero values.
(a); (b); A smooth and decomposable PC ( 2(b) ) and an equivalent Bayesian network ( 2(a) ). The Bayesian network is over 4 variables \mathbf{X}=\{X_{1},X_{2},X_{3},X_{4}\} and 2 hidden variables \mathbf{Z}=\{Z_{1},Z_{2}\} with h=2 hidden states. The feedforward computation order is from left to right; \bigodot are input Bernoulli distributions, \bigotimes are product units, and \bigoplus are sum units; parameter values are annotated in the box. The probability of each unit given input assignment \{X_{1}\!=\!0,X_{2}\!=\!1,X_{3}\!=\!0,X_{4}\!=\!1\} is labeled red.
(a) PC with fully connected layers; (b) PC after pruning operation; (c) PC after growing operation; A demonstration of the pruning and growing operation. From 3(a) to 3(b) , the red edges are pruned. From 3(b) to 3(c) , the nodes are doubled, and each parameter is copied 3 times.
(a) eParam removes the edge with \theta\!=\!0.1; (b) eFlow removes the edge with \theta\!=\!0.2; A case study comparing pruning heuristics ( eParam and eFlow ) on the PC in Figure 2 given sample \{X_{1}\!=\!0,X_{2}\!=\!1,X_{3}\!=\!0,X_{4}\!=\!1\} . The pruned edges are dashed and parameters are re-normalized.
Compared to the likelihood computed in Figure 2 , the changed likelihoods are in red, showing that pruning by flows results in less likelihood decrease.
(a) Comparison of heuristics eRand , eParam , and eFlow . Heuristic eFlow can prune up to 80% of the parameters without much loglikelihoods decrease.; (b) Histogram of parameters before (the same as in Figure 1 ) and after pruning. The parameter values take higher significance after pruning.; Empirical evaluation of the pruning operation.
Figure 6 : Comparing the actual loglikelihood drop ( \Delta LL) and eFlow heuristics (the approximated upper bound in Equation 3 ). The approximate bound matches closely to the actual loglikelihood drop.; Comparing the actual loglikelihood drop ( \Delta LL) and eFlow heuristics (the approximated upper bound in Equation 3 ). The approximate bound matches closely to the actual loglikelihood drop.
Figure 7 : Growing operation. Each unit is doubled, and each parameterized edge is copied 3 times: (n^{\mathtt{new}},c^{\mathtt{new}}) ( orange ), (n^{\mathtt{new}},c) ( purple ), and (n,c^{\mathtt{new}}) ( green ).; Growing operation. Each unit is doubled, and each parameterized edge is copied 3 times: (n^{\mathtt{new}},c^{\mathtt{new}}) ( orange ), (n^{\mathtt{new}},c) ( purple ), and (n,c^{\mathtt{new}}) ( green ).
Figure 8 : Model compression via pruning and finetuning. We report the training set bpd (y-axis) in terms of the number of parameters (x-axis) for different numbers of latent states. For each curve, compression starts from the right (initial PC #Params |\mathcal{C}^{\mathtt{init}}| ) and ends at the left (compressed PC #Params |\mathcal{C}^{\mathtt{com}}| ); compression rate (1 - |\mathcal{C}^{\mathtt{com}}| / |\mathcal{C}^{\mathtt{init}}| ) is annotated next to each curve.; Model compression via pruning and finetuning. We report the training set bpd (y-axis) in terms of the number of parameters (x-axis) for different numbers of latent states. For each curve, compression starts from the right (initial PC #Params |\mathcal{C}^{\mathtt{init}}| ) and ends at the left (compressed PC #Params |\mathcal{C}^{\mathtt{com}}| ); compression rate (1 - |\mathcal{C}^{\mathtt{com}}| / |\mathcal{C}^{\mathtt{init}}| ) is annotated next to each curve.
Figure 9 : Structure learning via 75% pruning, growing and finetuning. We report bpd (y-axis) on both train (red) and test set (green) in terms of the number of latent states (x-axis). For each curve, training starts from the top (large bpd) and ends at the bottom (small bpd).; Structure learning via 75% pruning, growing and finetuning. We report bpd (y-axis) on both train (red) and test set (green) in terms of the number of latent states (x-axis). For each curve, training starts from the top (large bpd) and ends at the bottom (small bpd).
Density estimation performance on MNIST-family datasets in test set bpd.
Character-level language modeling results on Penn Tree Bank in test set bpd.
Dataset statistics including number of variables ( #vars ), number of categories for each variable ( #cat ), and number of samples for training, validation and test set ( #train , #valid , #test ).