SlowGuess/ACL_2020_OCR huggingface dataset:
Here  $\lambda \in (0,1)$  is a linear interpolant for selecting between reconstructed embeddings and pre-learnt continuous embeddings. We initialize  $\lambda$  to 1 and gradually decrease  $\lambda$  as training proceeds. This enables the method to gradually rely more and more on reconstruction from discrete embeddings. We also add a regularization term  $||\mathcal{D}\circ \mathcal{R}(\mathbf{e}) - \mathbf{e}||_2^2$  during the training to encourage the reconstructed embeddings to match the pre-learnt continuous em
2We show in our experiments that this guidance, while helpful, is not always needed.
Dataset	Entities	Relations	Train	Valid	Test
FB15k	14,951	1,345	483,142	50,000	59,071
FB15k-237	14,541	237	272,115	17,535	20,466
WN18	40,943	18	141,442	5,000	5,000
WN18RR	40,943	11	86,835	3,034	3,134
Table 2: A summary of dataset statistics
beddings.
We compare the baseline continuous representations described earlier in Table 1 with four discrete representation learning techniques described in this paper:
VQ-CL:  $\mathcal{D} = \mathrm{VQ}$  and  $\mathcal{R} = \mathrm{CL}$
VQ-NL:  $\mathcal{D} = \mathrm{VQ}$  and  $\mathcal{R} = \mathrm{NL}$
- TS-CL:  $\mathcal{D} = \mathrm{TS}$  and  $\mathcal{R} = \mathrm{CL}$
- TS-NL:  $\mathcal{D} = \mathrm{TS}$  and  $\mathcal{R} = \mathrm{NL}$
We evaluate our approach on four standard link prediction datasets:
- FB15k (Bordes et al., 2013) is a subset of Freebase.
- FB15k-237 (Toutanova et al., 2015) is a subset of the FB15k dataset created by removing inverse relations that cause test leakage.
- WN18 (Bordes et al., 2013) is a subset of WordNet.
- WN18RR (Dettmers et al., 2018) is a subset of the WN18 dataset created by removing inverse relations.
We summarize all the data statistics in Table 2. We also use the Countries dataset (Bouchard et al., 2015) for some in-depth analysis of inference abilities of discrete representations.
We implement discrete KG representation learning by extending OpenKE (Han et al., 2018), an open-source framework for learning KG embeddings implemented on PyTorch  $^{3}$ . We train and test all our models on a single 2080Ti system. We set  $K = 32$  and  $D = 10$  in our experiments unless stated otherwise. For the linear embedding transformation function in the non-linear reconstruction approach, we use a hidden layer of 100 hidden units. We
set  $\lambda$  as  $\lambda = \frac{1}{\sqrt{t}}$  at the  $t^{th}$  epoch. We tune the regularization coefficient using grid search on the validation set.
Link Prediction: We learn discrete representations corresponding to various continuous KG representations (described in Table 1) and compare the obtained discrete representations with their continuous counterparts. We use the same hyper-parameter settings as in the original KG embedding papers. We generate  $n_e$  candidate triples for each test triple by combining the test entity-relation pair with all possible entities  $\mathcal{E}$ . We use the filtered setting (Bordes et al., 2013), i.e. all known true triples are removed from the candidate set except for the current test triple. We use standard evaluation metrics previously used in the literature: mean reciprocal rank (MRR) and hits@10 (H@10). Mean reciprocal rank is the average of the inverse of the mean rank assigned to the true triple over all candidate triples. Hits@10 measures the percentage of times a true triple is ranked within the top 10 candidate triples. In addition, in order to report the compression efficiency of the discrete representations, we also report the compression ratio which is computed as follows:
$$
C R = \frac {\text {S t o r a g e (c o n t i n u o u s)}}{\text {S t o r a g e (d i s c r e t e)}}
$$
Here, Storage(continuous) is the storage used to store full continuous KG representations. Storage(discrete) is the storage used in the discrete representation learning method (during the testing stage). This includes discrete KG representations as well as parameters of the reverse-discretization function (i.e. codebook or Bi-LSTM parameters).
Tables 3, 4, 5 and 6 show our results on the link prediction task on the four datasets respectively. In Table 3, we compare various continuous representations with the four discrete representation learning techniques described in this paper. We find that the discrete representations sustain only minor losses in performance (and are sometimes actually better than their continuous counterparts) in terms of both evaluation metrics: MRR and H@10, while being able to obtain significant embedding compression (42x-585x). Table 3 also compares the different discrete representation learning approaches. We observe that TS-NL which uses tempering softmax and non-linear reconstruction performs the best in most of the settings. This observation was also
<sup>3</sup>https://github.com/thunlp/OpenKE
Continuous	CR (CL)	VQ-CL	TS-CL	CR (NL)	VQ-NL	TS-NL
MRR	H@10	MRR	H@10	MRR	H@10	MRR	H@10	MRR	H@10
TransE	0.463	0.749	46.3	0.462	0.748	0.467	0.749	42.6	0.463	0.746	0.477	0.755
DistMult	0.798	0.893	77.6	0.750	0.859	0.775	0.864	71.4	0.756	0.868	0.790	0.882
HolE	0.524	0.739	112.6	0.515	0.708	0.517	0.711	103.8	0.517	0.717	0.525	0.726
ComplEx	0.692	0.840	262.3	0.651	0.802	0.653	0.814	228.4	0.670	0.818	0.678	0.833
ConvE	0.657	0.831	77.6	0.618	0.774	0.620	0.798	71.4	0.626	0.793	0.644	0.820
RotatE	0.797	0.884	585.3	0.765	0.840	0.782	0.876	495.2	0.789	0.878	0.798	0.881
HypER	0.790	0.734	177.5	0.743	0.706	0.754	0.715	161.1	0.758	0.718	0.763	0.726
TuckER	0.795	0.741	177.5	0.773	0.714	0.782	0.729	161.1	0.787	0.723	0.783	0.726
Table 3: Results of several models and our proposed discrete counterparts evaluated on the FB15K dataset
Continuous	Discrete (TS-NL)
MRR	H@10	CR	MRR	H@10
TransE	0.495	0.943	103.3	0.499	0.940
DistMult	0.797	0.946	143.2	0.774	0.921
HolE	0.938	0.949	228.6	0.938	0.929
ComplEx	0.941	0.947	437.1	0.934	0.936
ConvE	0.943	0.956	143.2	0.933	0.936
RotatE	0.949	0.959	952.6	0.946	0.952
HypER	0.951	0.947	327.9	0.946	0.942
TuckER	0.953	0.949	327.9	0.924	0.920
Table 4: Results of several models and our proposed discrete counterpart (TS-NL) evaluated on the WN18 dataset
Continuous	Discrete (TS-NL)
MRR	H@10	CR	MRR	H@10
TransE	0.294	0.465	43.1	0.298	0.463
DistMult	0.241	0.419	71.8	0.241	0.422
HolE	0.318	0.430	104.0	0.316	0.428
ComplEx	0.247	0.428	228.5	0.238	0.411
ConvE	0.325	0.501	71.8	0.321	0.488
RotatE	0.338	0.533	495.2	0.336	0.528
HypER	0.341	0.252	161.3	0.332	0.286
TuckER	0.358	0.266	161.3	0.331	0.279
made on the other three datasets. Hence, in Tables 4, 5 and 6, we only compare TS-NL with the continuous representations. We again observe that TS-NL compresses the KG embeddings (71x-952x) while suffering only a minor loss in performance.
Logical Inference with Discrete representations: KG embeddings give us a way to perform logical inference and reason about knowledge. In this experiment, we explore if discrete representations retain the ability to perform inference and reasoning in KGs. We evaluate our models on the countries dataset (Bouchard et al., 2015) which was designed to test the logical inference capabilities of KG embedding models. We use the same evaluation protocol as in (Nickel et al., 2016) for our
Table 5: Results of several models and our proposed discrete counterpart (TS-NL) evaluated on the FB15K-237 dataset.
Continuous	Discrete (TS-NL)
MRR	H@10	CR	MRR	H@10
TransE	0.226	0.501	105.2	0.230	0.498
DistMult	0.430	0.490	143.8	0.423	0.476
HolE	0.338	0.438	228.8	0.346	0.435
ComplEx	0.440	0.510	437.2	0.433	0.494
ConvE	0.430	0.520	143.8	0.431	0.500
RotatE	0.476	0.571	952.6	0.452	0.546
HypER	0.465	0.436	328.0	0.460	0.437
TuckER	0.470	0.443	328.0	0.452	0.442
Table 6: Results of several models and our proposed discrete counterpart (TS-NL) evaluated on the WN18RR dataset
experiments. The countries dataset contains 2 relations and 272 entities (244 countries, 5 regions and 23 subregions) and 3 tasks are posed, requiring subsequently longer and harder inference than the previous one:
1. Task S1 poses queries of the form locatedIn(c; ), and the answer is one of the five regions.
2. Task S2 poses queries of the form neighborOf(c1; c2)  $\wedge$  locatedIn(c2; r)  $\Longrightarrow$  locatedIn(c1; r)
3. Task S3 poses queries of the form neighborOf(c1; c2)  $\wedge$  locatedIn(c2; s)  $\wedge$  locatedIn(s; r)  $\Longrightarrow$  locatedIn(c1; r):
We use the AUC-PR metric, which was also used in previous works (Bouchard et al., 2015; Nickel et al., 2016). Table 7 shows our results. We find that TS-NL is a very good KG representation for KG inference. Infact, we find that TS-NL outperforms many of their continuous counterparts.
Additional Inference Cost: A tradeoff in learning discrete KG representations is that the inference time increases as we need to decompress discrete representations into continuous embeddings for every entity before using them by looking up the
S1	S2	S3
Ct.	Dis.	Ct.	Dis.	Ct.	Dis.
TransE	0.93	0.95	0.56	0.59	0.34	0.41
DistMult	1.00	0.97	0.72	0.71	0.52	0.55
HolE	1.00	0.97	0.77	0.80	0.70	0.74
ComplEx	0.97	0.97	0.57	0.56	0.43	0.45
ConvE	1.00	1.00	0.99	0.96	0.86	0.87
RotatE	1.00	1.00	1.00	0.98	0.95	0.91
HypER	1.00	0.97	0.76	0.80	0.68	0.75
TuckER	1.00	1.00	0.85	0.88	0.75	0.79
codebook or running inference on the LSTM reconstruction model. In practice, we found that this additional inference cost was very small. For example, the additional inference cost of running TransE on the entire FB15K test set was  $\approx 1$  minute for codebook lookup and  $\approx 2.5$  minutes for non-linear reconstruction approach on our single 2080Ti system. The additional inference cost for the other continuous KG representations were similarly low.
Varying  $K$  and  $D$ : There is an evident tradeoff between the extent of compression (which is dictated by the choice of K and D) and model performance. In order to explore this tradeoff, we plot heatmaps of performance (MRR) and compression ratio (CR) on the FB15K test set as we vary K and D for TransE in Figure 1. Not surprisingly, the performance drops as the compression increases. Plotting these heat maps would allow the end user to pick K and D depending on their tolerance to loss in performance.
Dependence on guidance from continuous embeddings: We evaluate the contribution of the guidance from continuous embeddings in learning discrete KG representations. Figure 2 compares the test MRR for TS-NL as training proceeds on the FB-15K dataset when we do or do not have guidance from the continuous representation (TransE). We observe that learning in the unguided model is much slower than the guided model. However, the guided model achieves almost similar performance in the end. Thus, we conclude that while guidance helps us achieve faster and more stable convergence, it is not necessary to learn discrete representations.
Quality of the Discrete representations: We also assess the quality of the learnt discrete entity representations directly as features for the link predic
Figure 1: Heatmaps of performance (MRR) and CR for TS-NL on FB15K dataset as we vary K and D - darker is better.
Figure 2: Test MRR for TS-NL as training proceeds on FB-15K dataset with and without guidance from continuous embeddings.
Table 7: Results of several continuous representations (ct.) and discrete TS-NL (dis.) evaluated on the three tasks (S1, S2 and S3) of logical inference on countries dataset.
MRR	H@10
TransE	0.472	0.740
DistMult	0.778	0.870
HolE	0.497	0.706
ComplEx	0.654	0.816
ConvE	0.596	0.797
RotatE	0.743	0.828
HypER	0.722	0.694
TuckER	0.742	0.705
Table 8: Transfer results on FB15K dataset.
Code	Entities
WN	3-7-0-6-X	animalize, workAnimal, farmAnimal, animal_husbandry, offspring, animal, invertebrate, marineAnimal, animal_kingdom, predator
5-3-0-X-1	jabalpur, calcutta, bombay, hyderabad, chennai, lucknow, mysore
FB	2-X-7-4-1	novelist, dramatist, actor, writer, cartoonist, poet, songwriter, musician
2-5-X-4-1	albert_einstein, voltaire, isaac_newton, nikola_tesla
Table 9: Example learned codes (K=8, D=5, X ∈ {0,7}) for Freebase (FB) and Wordnet (WN). Similar entities are assigned to close-by codes.
tion task. In this case, we only retain the discrete entity representations learnt by TS-NL and learn a new LSTM based non-linear reverse-discretization on the validation set. Then, we obtain the link-prediction performance on the test set as before (see Table 8 for transfer results on the FB15K dataset). We observe that the performance of this "transfer" model is close to that of the original model which used a pre-trained reverse-discretization model (compare Table 8 with the shaded part of Table 3). Note that, in the "transfer" setting, we can achieve much higher compression as we do not even need to store the reverse-discretization model.
Interpretability of discrete representations: The discrete codes provide us with additional interpretability which continuous representations can lack. In Table 9, we show a sample of learned codes for the two datasets. We observe that semantically similar entities are assigned to close-by codes.
Deep learning model compression has attracted many research efforts in the last few years (Han et al., 2015). These efforts include network pruning (Reed, 1993; Castellano et al., 1997), weight sharing (Ullrich et al., 2017), quantization (Lin et al., 2016), low-precision computation (Hwang and Sung, 2014; Courbariaux et al., 2015) and knowledge distillation (Hinton et al., 2015). These techniques can also be used for embedding compression. Press and Wolf (2016) and Inan et al. (2016) propose weight-tying approaches that learn input and output representations jointly. Matrix factorization-based methods (Acharya et al., 2019; Shu and Nakayama, 2017; Li et al., 2018) have also been proposed which approximate an embedding matrix with smaller matrices or clusters. Closest to our work are (Shu and Nakayama, 2017; Chen et al., 2018; Chen and Sun, 2019) who present similar approaches to learn discrete codings for word embeddings using multiple codebooks, i.e. product quantization (Jegou et al., 2010). Similar
techniques have used been used by van den Oord et al. (2017) who extend VAEs to learn discrete representations using vector quantization in the image domain. This allows the VAE model to circumvent its well known issues of "posterior collapse". All these previous works have been applied to the image domain, and sometimes in language to learn discrete word embeddings. In this work, we present the first results on compressing KG embeddings and also show how the compressed embeddings can be used to support various knowledge based applications such as KG inference.
The embedding layer contains majority of the parameters in any representation learning approach on knowledge graphs. This is a barrier in successful deployment of models using knowledge graphs at scale on user-facing computing devices. In this work, we proposed novel and general approaches for KG embedding compression. Our approaches learn to represent entities in a KG as a vector of discrete codes in an end-to-end fashion. At test time, the discrete KG representation can be cheaply and efficiently converted to a dense embedding and then used in any downstream application requiring the use of a knowledge graph. We evaluated our proposed methods on different link prediction and KG inference tasks and show that the proposed methods for KG embedding compression can effectively compress the KG embedding table without suffering any significant loss in performance. In this work, we only considered the problem of learning discrete entity representations. In the future, we would like to jointly learn discrete representations of entities as well as relations.
MS would like to thank the anonymous reviewers, along with Karen Livescu, Kevin Gimpel and Shun- ing Jin for their valuable comments and suggestions on this work.
Anish Acharya, Rahul Goel, Angeliki Metallinou, and Inderjit Dhillon. 2019. Online embedding compression for text classification using low rank matrix factorization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6196-6203.
Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853.
Ivana Balažević, Carl Allen, and Timothy M. Hospedales. 2019a. Hypernetwork knowledge graph embeddings. In International Conference on Artificial Neural Networks, pages 553-565. Springer.
Ivana Balažević, Carl Allen, and Timothy M. Hospedales. 2019b. Tucker: Tensor factorization for knowledge graph completion. arXiv preprint arXiv:1901.09590.
Dana H Ballard. 1997. An introduction to natural computation. MIT Press.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795.
Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning structured embeddings of knowledge bases. In Twenty-Fifth AAAI Conference on Artificial Intelligence.
Guillaume Bouchard, Sameer Singh, and Theo Trouillon. 2015. On approximate reasoning capabilities of low-rank vector spaces. In 2015 AAAI Spring Symposium Series.
Hongping Cai, Fei Yan, and Krystian Mikolajczyk. 2010. Learning weights for codebook in image classification and retrieval. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 2320-2327. IEEE.
Giovanna Castellano, Anna Maria Fanelli, and Marcello Pelillo. 1997. An iterative pruning algorithm for feedforward neural networks. IEEE transactions on Neural networks, 8(3):519-531.
Lawrence Cayton. 2008. Fast nearest neighbor retrieval for bregman divergences. In Proceedings of the 25th international conference on Machine learning, pages 112-119. ACM.
Ting Chen, Martin Renqiang Min, and Yizhou Sun. 2018. Learning k-way d-dimensional discrete codes for compact embedding representations. arXiv preprint arXiv:1806.09464.
Ting Chen and Yizhou Sun. 2019. Differentiable product quantization for end-to-end embedding compression.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015. Low precision arithmetic for deep learning. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Thirty-Second AAAI Conference on Artificial Intelligence.
Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
Xu Han, Shulin Cao, Lv Xin, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018. Openke: An open toolkit for knowledge embedding. In Proceedings of EMNLP.
He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. arXiv preprint arXiv:1704.07130.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Kyuyeon Hwang and Wonyong Sung. 2014. Fixed-point feedforward deep neural network design using weights +1, 0, and -1. In SiPS, pages 174-179. IEEE.
Hakan Inan, Khashayar Khosravi, and Richard Socher. 2016. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462.
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-128.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 687-696.
Seyed Mehran Kazemi and David Poole. 2018. Simple embedding for link prediction in knowledge graphs. In Advances in Neural Information Processing Systems, pages 4284-4295.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence.
Zhongliang Li, Raymond Kulhanek, Shaojun Wang, Yunxin Zhao, and Shuang Wu. 2018. Slim embedding layers for recurrent neural language models. In Thirty-Second AAAI Conference on Artificial Intelligence.
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. 2016. Fixed point quantization of deep convolutional networks. In International Conference on Machine Learning, pages 2849-2858.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference on artificial intelligence.
Vanessa Lopez, Christina Unger, Philipp Cimiano, and Enrico Motta. 2013. Evaluating question answering over linked data. Web Semantics Science Services And Agents On The World Wide Web, 21:3-13.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2016. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712.
Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016. Holographic embeddings of knowledge graphs. In Thirtieth Aaai conference on artificial intelligence.
Aaron van den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pages 6306-6315.
Ofir Press and Lior Wolf. 2016. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859.
Russell Reed. 1993. Pruning algorithms-a survey. IEEE transactions on Neural Networks, 4(5):740-747.
Raphael Shu and Hideki Nakayama. 2017. Compressing word embeddings via deep compositional code learning. arXiv preprint arXiv:1711.01068.
Amit Singhal. 2012. Introducing the knowledge graph: things, not strings. 2012 (accessed: 16 May 2012).
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems, pages 926-934.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pages 2071-2080.
Karen Ullrich, Edward Meeds, and Max Welling. 2017. Soft weight-sharing for neural network compression. arXiv preprint arXiv:1702.04008.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Twenty-Eighth AAAI conference on artificial intelligence.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575.
Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Aug-mentation end-to-end dialogue systems with commonsense knowledge. In Thirty-Second AAAI Conference on Artificial Intelligence.