How to compute the log_prob of a weighted random sample without replacement

username: kristen
cooked: <p>I’m looking for something like <code>torch.distributions.Multinomial().log_prob()</code>, that only allows for 1 sample from each category.</p>
<p>I know you can do weighted random sampling without replacement using <a href="https://pytorch.org/docs/stable/generated/torch.multinomial.html#torch.multinomial" rel="noopener nofollow ugc">torch.multinomial</a>, and I believe that is sampling from the categorical distribution. I’m trying to figure out how to compute the log_prob of such an unordered weighted sample? (Not sure if I compute it as multiple independent samples from a categorical distribution, or if it should be modeled as a conditional bernoulli distribution, or hypergeometric etc.)</p>
<p>This is perhaps more of a statistics theory question, but just wanted to check it’s not already implemented before I go and try to do it myself.</p>

username: KFrank
cooked: <p>Hi Kristen!</p>
<aside class="quote no-group" data-username="kristen" data-post="1" data-topic="110643" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/c4cdca/48.png" class="avatar"> kristen:</div>
<blockquote>
<p>I’m looking for something like <code>torch.distributions.Multinomial().log_prob()</code>, that only allows for 1 sample from each category.</p>
</blockquote>
</aside>
<p>I’m neither a statistics expert, nor am I sure that I fully understand what you<br>
are asking.  Nonetheless I am cheerfully willing to provide misinformation to<br>
this forum.</p>
<p>I think that for a <em>single</em> sample, you get the same distribution, and hence the<br>
same <code>log_prob()</code>, from, say, <code>Multinomial</code> without replacement as you do<br>
with replacement.  With replacement, individual samples are independent.</p>
<p>Without replacement, you now get a non-trivial <em>joint</em> distribution – for example,<br>
without replacement, <code>P (4, 4) = 0</code> (because you can’t sample the value <code>4</code><br>
a second time unless you replace it after the first sample).</p>
<p>Best.</p>
<p>K. Frank</p>

username: googlebot
cooked: <p>You’d need to adjust the combinatorial term in multinomial pmf formula, filtering out x[i]&gt;1 cases, but you don’t have to do that to train p parameters on observed x vectors - as pmf ratios would be the same as for multinomial, and omission of a normalization constant won’t affect gradient directions.</p>

username: kristen
cooked: <p>Thanks <a class="mention" href="/u/kfrank">@KFrank</a> and <a class="mention" href="/u/googlebot">@googlebot</a>. That makes sense (even though my question did not <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> )</p>

username: googlebot
cooked: <p>Looking at this again, my answer may be incorrect - it assumes that we observe censored draws made with static p; if instead p is inflated on subsequent draws as categories get removed, marginal probability ratios will differ… That’s indeed something similar to multivariate hypergeometric (urn model), but for continuous numbers…</p>

username: kristen
cooked: <p>No worries. I was expecting what I described to be a named/known probability distribution, but it turns out it’s not.</p>