Maximum mean discrepancy (MMD) and radial basis function (rbf)

lliu25: <p>what is a concise and correct way to implement rbf and MMD, considering two vectors? Can rbf function be calculated directly by using torch.norm?</p>

Jordan_Campbell: <p>My implementation to compute the MMD between two sets of samples:</p>
<p>Here x and y are batches of images with shape <code>[B,1,W,H]</code></p>
<pre><code>	x = x.view(x.size(0), x.size(2) * x.size(3))
	y = y.view(y.size(0), y.size(2) * y.size(3))

	xx, yy, zz = torch.mm(x,x.t()), torch.mm(y,y.t()), torch.mm(x,y.t())

	rx = (xx.diag().unsqueeze(0).expand_as(xx))
	ry = (yy.diag().unsqueeze(0).expand_as(yy))

	K = torch.exp(- self.alpha * (rx.t() + rx - 2*xx))
	L = torch.exp(- self.alpha * (ry.t() + ry - 2*yy))
	P = torch.exp(- self.alpha * (rx.t() + ry - 2*zz))

    beta = (1./(B*(B-1)))
    gamma = (2./(B*B)) 

	return beta * (torch.sum(K)+torch.sum(L)) - gamma * torch.sum(P)
</code></pre>
<p>Credit goes to <a class="mention" href="/u/fmassa">@fmassa</a> for <a href="https://discuss.pytorch.org/t/build-your-own-loss-function-in-pytorch/235/7?u=jordan_campbell">this</a> previous answer.</p>
<p>You can easily change the kernel by changing the definition for K,L,P.</p>

lliu25: <p>Thank you so much for the fast reply! In this implementation, I guess x and y do not need to have the same B (batch size)? Or B needs to be the same?</p>

Jordan_Campbell: <p>With this implementation your batches will have to be the same size, since <code>rx.t()</code> and <code>ry</code> are square matrices and will end up with different sizes if batch sizes are different (and then <code>zz</code> also won’t be square).</p>
<p>The best option is just to randomly sample <code>N</code> elements from each of your batches so they are the same size.</p>

lliu25: <p>I see. Thank you again for the explanation.</p>

noeagles: <p>You may want to check <a href="https://github.com/josipd/torch-two-sample" rel="nofollow noopener">https://github.com/josipd/torch-two-sample</a> a pytorch based library.</p>

Asif_Hossain: <p><a class="mention" href="/u/jordan_campbell">@Jordan_Campbell</a> , what is the value of alpha here? I mean what could be the suitable and general value of alpha. One more thing, in my case, batch size, B =1. so, what will be the value of beta ?<br>
thanks in advance</p>

ryo: <aside class="quote no-group" data-username="Asif_Hossain" data-post="7" data-topic="1875">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asif_hossain/48/2979_2.png" class="avatar"> Asif_Hossain:</div>
<blockquote>
<p>One more thing, in my case, batch size, B =1.</p>
</blockquote>
</aside>
<p>I’m not sure comparing distributions makes sense for a single sample.</p>

checkadelic: <p>I found this code when reading the thread:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/ZongxianLee/MMD_Loss.Pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/30426652?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/ZongxianLee/MMD_Loss.Pytorch" target="_blank" rel="nofollow noopener">ZongxianLee/MMD_Loss.Pytorch</a></h3>

<p>A pytorch implementation of Maximum Mean Discrepancies(MMD) loss - ZongxianLee/MMD_Loss.Pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>