text: 155,96,1284151359491788800,1069184356533583872,Ekaterina Lobacheva,"Our new paper On Power Laws in Deep Ensembles is on arXiv: <LINK> Credits to @nadiinchi, Maxim Kodryan, Dmitry Vetrov @bayesgroup We investigate asymptotic properties of CNLL as a function of ensemble size n, network size s, and the number of parameters B. 1/4 <LINK> CNLL and NLL of deep ensemble follow power law w.r.t. ensemble size n: CNLL_n = c + b n^a Moreover, CNNL follows power law w.r.t. network size s and the total number of parameters B. 2/4 <LINK> Memory Split Advantage effect: our practically important funding is that one large network may perform worse than an ensemble of several medium-size networks with the same total number of parameters. 3/4 <LINK> Given relatively small number of trained networks, we can use the discovered power laws to predict: - NLL and CNLL of large ensembles - optimal memory split given a memory budget. 4/4 <LINK>",http://arxiv.org/abs/2007.08483,"Ensembles of deep neural networks are known to achieve state-of-the-art performance in uncertainty estimation and lead to accuracy improvement. In this work, we focus on a classification problem and investigate the behavior of both non-calibrated and calibrated negative log-likelihood (CNLL) of a deep ensemble as a function of the ensemble size and the member network size. We indicate the conditions under which CNLL follows a power law w.r.t. ensemble size or member network size, and analyze the dynamics of the parameters of the discovered power laws. Our important practical finding is that one large network may perform worse than an ensemble of several medium-size networks with the same total number of parameters (we call this ensemble a memory split). Using the detected power law-like dependencies, we can predict (1) the possible gain from the ensembling of networks with given structure, (2) the optimal memory split given a memory budget, based on a relatively small number of trained networks. We describe the memory split advantage effect in more details in arXiv:2005.07292 ",On Power Laws in Deep Ensembles,4,"['Our new paper On Power Laws in Deep Ensembles is on arXiv: <LINK>\nCredits to @nadiinchi, Maxim Kodryan, Dmitry Vetrov @bayesgroup\n\nWe investigate asymptotic properties of CNLL as a function of ensemble size n, network size s, and the number of parameters B. 1/4 <LINK>', 'CNLL and NLL of deep ensemble follow power law w.r.t. ensemble size n: CNLL_n = c + b n^a\nMoreover, CNNL follows power law w.r.t. network size s and the total number of parameters B.\n2/4 https://t.co/HO8aeVCgHj', 'Memory Split Advantage effect: our practically important funding is that one large network may perform worse than an ensemble of several medium-size networks with the same total number of parameters. 3/4 https://t.co/g3fD9ZWk7c', 'Given relatively small number of trained networks, we can use the discovered power laws to predict:\n- NLL and CNLL of large ensembles \n- optimal memory split given a memory budget. 4/4 https://t.co/pFK2gumGPf']",20,07,863