Simple Optimization/Embedding Example

rajan: <p>I’m looking to do a simple optimization, similar to the collaborative learning example that Jeremy Howard (fast.ai) showed in his Deep Learning MOOC (Lesson 4 @ 1:08)</p>
<p>Given <code>blockData.shape=(20,14)</code> filled with random numbers, I want to start with two matrices also filled with random numbers, with shapes <code>vert.shape=(20,10)</code> and <code>hori.shape=(10,14)</code> such that at the end of the optimization I minimize the quantity:</p>
<pre><code class="lang-auto">mse( blockData - dot(vert,hori) )
</code></pre>
<p>by modifying hori and vert simultaneously using autograd.</p>
<p>If I write the <code>backward()</code> call</p>
<pre><code class="lang-auto">loss = mse(tensor(numpy.dot(vert,hori)),blockData)
loss.backward()
</code></pre>
<p>I get an error <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code> which, based on my internet searching, is most commonly seen in cases when the <code>.requires_grad</code> isn’t set to True.</p>
<p>It appears I have a case where the gradient isn’t computed because (maybe) there isn’t a variable to set <code>requires_grad=True</code> for.</p>
<p>Two main questions are:</p>
<ol>
<li>How can I display the contents of <code>tensors</code> so I can see what element 0 is referring to?</li>
<li>Can autograd do what I’m asking it to? And if so, how?</li>
</ol>
<hr>
<p>Here is the code I have, which isn’t what I’m looking for, but is close, and it runs without error:</p>
<pre><code class="lang-auto">from fastai.basics import tensor, nn
import torch, numpy, pandas

def hypothesis(vert,hori):
    return numpy.dot(vert,hori)

def mse(y_hat,y):
    return ((y_hat-y)**2).mean()

def update(y_hat):
    # perform gradient descent
    loss = mse(y_hat,blockData)
    loss.backward()
    if t%10 == 0:
        print(t,'-------------',loss)
    with torch.no_grad():
        y_hat.sub_(lr * y_hat.grad)
        y_hat.grad.zero_()

vecSize = 10
shape = (20,14)
# random large block of data
blockData = tensor(numpy.random.random_sample(shape))

hori = nn.Parameter(tensor(numpy.random.random_sample((vecSize,shape[1]))))
vert = nn.Parameter(tensor(numpy.random.random_sample((shape[0],vecSize))))


lr = 1e-1
y_hat = tensor(hypothesis(vert,hori))
y_hat.requires_grad_(True)
for t in range(101):
    update(y_hat)
</code></pre>
<p>The code above will drive <code>y_hat</code> to the correct answer, but that’s not really the idea, since what I’m trying to do is to get <code>vert</code> and <code>hori</code> to drive towards the correct answers as part of the same optimization. As it is written now I’m just taking the gradient from the hypothesis matrix (<code>y_hat</code>) to the target (<code>blockData</code>), which is a trivial optimization.</p>

ptrblck: <p>I’m not familiar with the mentioned Lesson in FastAI, however from a code perspective you are detaching the computation graph from <code>vert</code> and <code>hori</code> by using <code>numpy.dot</code> and rewrapping the result in a tensor with <code>requires_grad=True</code>.<br>
Since <code>vert</code> and <code>hori</code> are already <code>nn.Parameters</code>, you could stick to PyTorch methods, so that Autograd will automatically create the computation graph (and can thus backpropagate).</p>
<p>I’m not sure, if this will yield the desired result, but this core updates <code>vert</code> and <code>hori</code>:</p>
<pre><code class="lang-python">def hypothesis(vert,hori):
    return torch.matmul(vert,hori)

def mse(y_hat,y):
    return ((y_hat-y)**2).mean()

def update(y_hat):
    y_hat = hypothesis(vert,hori)
    # perform gradient descent
    loss = mse(y_hat,blockData)
    loss.backward()
    if t%10 == 0:
        print(t,'-------------',loss)
    with torch.no_grad():
        vert.sub_(lr * vert.grad)
        hori.sub_(lr * hori.grad)
        vert.grad.zero_()
        hori.grad.zero_()

vecSize = 10
shape = (20,14)
# random large block of data
blockData = torch.randn(shape)

hori = nn.Parameter(torch.randn(vecSize,shape[1]))
vert = nn.Parameter(torch.randn(shape[0],vecSize))

lr = 1e-1
for t in range(101):
    update(y_hat)
</code></pre>

rajan: <p>Thanks for the clear explanation, that fixed the issue (and gave me some useful things to read up on to dive deeper into Torch). The optimization works now, here is the full code. The output of the resultant tensors/vectors at the end isn’t probably the best method, but it works.</p>
<p>One minor note about the code below, I changed <code>vert</code> to <code>vertLatents</code> and <code>hori</code> to <code>horiLatents</code></p>
<pre><code class="lang-auto">from fastai.basics import tensor, nn
import torch, numpy, pandas

def hypothesis(vertLatents,horiLatents):
    return torch.matmul(vertLatents,horiLatents)

def mse(horiLatents,vertLatents,y):
    return ((torch.matmul(horiLatents,vertLatents)-y)**2).mean()

def update(horiLatents,vertLatents):
    # perform gradient descent
    loss = mse(vertLatents,horiLatents,blockData)
    loss.backward()
    if t%100 == 0:
        print(t,'-------------',loss)
    with torch.no_grad():
        vertLatents.sub_(lr * vertLatents.grad)
        vertLatents.grad.zero_()
        horiLatents.sub_(lr * horiLatents.grad)
        horiLatents.grad.zero_()
    return loss.item()

vecLatents = 10
shape = (20,14)
# random large block of data
blockData = tensor(numpy.random.random_sample(shape))

horiLatents = \
        nn.Parameter(tensor(numpy.random.random_sample((vecLatents,shape[1]))))
vertLatents = \
        nn.Parameter(tensor(numpy.random.random_sample((shape[0],vecLatents))))

lr = 1e-1
horiLatents.requires_grad_(True)
vertLatents.requires_grad_(True)
lossDict = {}
for t in range(10001):
    lossDict[t] = update(horiLatents,vertLatents)
pandas.DataFrame.from_dict(lossDict,orient='index').to_csv('lossDict.csv',
                                                           index=False)
pandas.DataFrame(horiLatents.data.tolist()).to_csv('horiLatents.csv',
                                                   index=False)
pandas.DataFrame(vertLatents.data.tolist()).to_csv('vertLatents.csv',
                                                   index=False)
pandas.DataFrame(blockData.data.tolist()).to_csv('blockData.csv',
                                                   index=False)
pandas.DataFrame(torch.matmul(vertLatents,horiLatents).data.tolist()).to_csv(
    'optimized.csv',index=False)
</code></pre>