Vandermonde Determinant Function

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I’ve been looking for an equivalent determinant function that computes the determinant of a Vandermonde Matrix. In short, the Vandermonde determinant scales better than a standard determinant due to the fact that the matrix is restricted.</p>
<p>Such a determinant is defined as,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35d00ab20b9cb11c0c82fd40e0ccade20c208633.png" alt="image" data-base62-sha1="7G324lOzTqUYEWDt7e1qkw96IFR" width="277" height="74"><br>
where <code>x</code> is a vector of length <code>n</code>.</p>
<p>I’ve written up a quick function that works, which is,</p>
<pre><code class="lang-auto">def vander_det(x: Tensor) -&gt; Tensor:
  """"
  :param x: A Tensor of size [B,N]
  """
  det = 1.
  for j in reversed(range(x.shape[-1])):
    for i in reversed(range(j)):
      det *= (x[:,j] - x[:,i])
  return det
</code></pre>
<p>Although this works I’ve noticed that it actually scales worse than using <code>torch.linalg.det</code> (most likely due to the use a nested-loop. I was wondering if 1) There already exists a vandermonde-determinant like function within PyTorch (which I’ve missed) or 2) Is there a way to parallelize the nested loop?</p>
<p>Any help would be greatly apprecitated!</p>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: KFrank
cooked: <p>Hi Alpha!</p>
<aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="1" data-topic="140179" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>I’ve noticed that it actually scales worse than using <code>torch.linalg.det</code></p>
</blockquote>
</aside>
<p>I doubt that your nested-loop version <em>scales</em> worse that <code>det()</code>.  Your<br>
version scales in time as <code>n**2</code> (just count the number of times through<br>
the loop), while <code>det()</code> scales as <code>n**3</code>.  Of course for moderate values<br>
of <code>n</code> your python-loop version is indeed likely to be much slower than<br>
<code>det()</code> because of the python, non-pipelined overhead.</p>
<aside class="quote no-group">
<blockquote>
<p>Is there a way to parallelize the nested loop?</p>
</blockquote>
</aside>
<p>Yes.  With a little bit of monkeying around, you can perform the <code>n**2</code><br>
computation with pure, non-loop tensor operations:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt;
&gt;&gt;&gt; n = 5
&gt;&gt;&gt;
&gt;&gt;&gt; x = torch.randn (n)   # random x
&gt;&gt;&gt;
&gt;&gt;&gt; V = x.unsqueeze (1).expand (n, n)**torch.arange (n)   # vandermonde matrix
&gt;&gt;&gt;
&gt;&gt;&gt; torch.det (V)   # vandermonde determinant
tensor(2.2679)
&gt;&gt;&gt;
&gt;&gt;&gt; d = x - x.unsqueeze (1)   # "outer difference" -- x_j - x_i
&gt;&gt;&gt;
&gt;&gt;&gt; (torch.ones (n, n).tril() + d.triu (1)).prod()   # product of upper triangle of d  =  det (V)
tensor(2.2679)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>!</p>
<p>Thanks for the quick and detailed response!</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="140179">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I doubt that your nested-loop version <em>scales</em> worse that <code>det()</code>. Your<br>
version scales in time as <code>n**2</code> (just count the number of times through<br>
the loop), while <code>det()</code> scales as <code>n**3</code>.</p>
</blockquote>
</aside>
<p>That’s what I thought but when I tested it the Vandermonde determinant scales bettter initially but for larger <code>N</code> performed worse. The Vandermonde det scaled approximately <code>N**2</code> but <code>torch.linalg.det</code> scaled better which doesn’t make sense to me because it’s an <code>N**3</code> operation, although that’s the naive scaling of it as it can be reduced.</p>
<p>What I’m probably doing is calculating the scaling wrong. The TL;DR of that is I just use <code>curve_fit</code> from <code>scipy</code> and fit a function of <code>f = a*x**b</code>. For Vandermonde is was about 2.1 and for <code>torch.linalg.det</code> it was like 1.4 (which doesn’t make sense at all, most likely because I’m calculating the scaling wrong). Perhaps it’d be better to <code>curve_fit</code> to logarithm of the runtime.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="140179">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Yes. With a little bit of monkeying around, you can perform the <code>n**2</code><br>
computation with pure, non-loop tensor operations:</p>
</blockquote>
</aside>
<p>Thanks for the great example!</p>