Deep learning: How to write pseudo algorithm for network architecture

username: aaditya
cooked: <p>How do I write pseudo algorithm for any deep learning model?</p>
<p>I was going through few deep learning papers on graph and there is pseudo algorithm for network architecture, Example :</p>
<pre><code class="lang-auto">Algorithm 3 SAGE Model [11]
Require: Aˆ is a normalization of A
1: function SageModel(Aˆ, X, L)
2: Z ← X
3: for i = 1 to L do
4: Z ← σ(

Z AZˆ

W (i)
)
5: Z ← L2NormalizeRows(Z)
6: return Z
</code></pre>
<p>That is overall architecture described in paper.</p>
<p>I want to write one pseudo algorithm for one custom network, Network structure is very simple, It’s sentence classification using dynamic rnn.</p>
<pre><code class="lang-auto">    Input is sentence [ batch_size x max_sentence_length x embedding_dim ]
    
    labels are [ batch_size ]
</code></pre>
<p>network is send sentences to rnn and get the feature vectors after send this feature vector to one other nano network for attention part and get attention vector, after getting attention vector add one dense layer to reshape the feature vectors and get the probability values shape [ batch_size ]</p>
<p>So nano network is:</p>
<pre><code class="lang-auto">    import numpy as np
    #simple soft attention
    
    def nano_network( logits, lstm_units ):
        
        # just for example 
        logits_ = tf.reshape(logits,[-1, lstm_units])
        attention_size = tf.get_variable(name='attention_size',
                                             shape=[lstm_units,1],
                                             dtype=tf.float32,
                                             initializer=tf.random_uniform_initializer(-0.01,0.01))
        
        attention_matmul =  tf.matmul(logits_,attention_size)
        output_reshape = tf.reshape(attention_matmul,[tf.shape(logits)[0],tf.shape(logits)[1],-1])
        return tf.squeeze(output_reshape)
</code></pre>
<p>Simple rnn network :</p>
<pre><code class="lang-auto">    import tensorflow as tf
    
    #simple network
    
    class Base_model(object):
    
        def __init__(self):
    
            tf.reset_default_graph()
    
            # define placeholders
            self.sentences        = tf.placeholder(tf.float32, [12, 50, 10], name='sentences') # batch_size x max_sentence_length x dim
            self.targets          = tf.placeholder(tf.int32, [12], name='labels'  )
            
            
            with tf.variable_scope('dynamic_rnn') as scope:
                cell = tf.nn.rnn_cell.LSTMCell(num_units=5, state_is_tuple=True)
                outputs, _states = tf.nn.dynamic_rnn(cell, self.sentences, dtype=tf.float32)
                
            
            #attention function
            self.output_s = nano_network(outputs,5)
            
            
            # simple linear projection
            self.output = tf.layers.dense(self.output_s,2)
            
            cross_entropy  = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=self.targets,logits=self.output)
            
            #loss calculation
            loss = tf.reduce_mean(cross_entropy)
    
            #train / network weights update
            self.train = tf.train.AdamOptimizer().minimize(loss)
            
            
            self.out = { 'loss':loss, 'train':self.train }
</code></pre>
<p>If someone want to run this code, here is random values for testing the code:</p>
<pre><code class="lang-auto">    # #model train
    
    # def rand_exec(model):
    #     with tf.Session() as sess:
    #         sess.run(tf.global_variables_initializer())
            
    #         for i in range(100):
    #             loss_ = sess.run(model.out,
    #                      feed_dict = {
    #                          model.sentences  : np.random.randint(0, 10, [12,50,10]),
    #                          model.targets  : np.random.randint(0, 2, [12] )})
                
    #             print(loss_['loss'])
                        
    # # wdim, hdim, vocab_size, num_labels,threshold,relation_embeddings,relation_dim,t,adj_file
    # if __name__ == '__main__':
    
    #     model = Base_model()
    #     out = rand_exec(model)
</code></pre>
<p>Now, I want to write pseudo algorithm for this simple network.</p>
<p>I tried to convert this network architecture into pseudo algorithm, Here is my pseudo code:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b19951f0862831415c2fcb7c1ab9f9e74fdbb274.png" data-download-href="https://discuss.pytorch.org/uploads/default/b19951f0862831415c2fcb7c1ab9f9e74fdbb274" title="37%20AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b19951f0862831415c2fcb7c1ab9f9e74fdbb274_2_650x500.png" alt="37%20AM" data-base62-sha1="pl6ZWdZgh2hamXRU6KjlvGwTQ5C" width="650" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b19951f0862831415c2fcb7c1ab9f9e74fdbb274_2_650x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b19951f0862831415c2fcb7c1ab9f9e74fdbb274_2_975x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/b19951f0862831415c2fcb7c1ab9f9e74fdbb274.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">37%20AM</span><span class="informations">979×753 64.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>[<a href="https://docs.google.com/document/d/1PkAlJbLh5U3nTYZiKgmdf2bkKa0gFwFA62TpGfnpo_s/edit?usp=sharing" rel="noopener nofollow ugc">Here is textual format of this algorithm</a>][4]</p>
<p>But I am confuse if it is correct or not like the parameters update rule and other things in algorithm.</p>
<p>I would greatly appreciate it if anyone kindly give me some advice on how to write pseudo algorithm  for this network architecture.</p>
<p>I know this is not pytorch code but I am not getting help from anywhere and since it’s more deep learning question than tensorflow question, that’s why I posted here.</p>
<p>Thank you !</p>

username: Balamurali_M
cooked: <p>The pseudo code looks good. The pseudo code describes the whole procedure you are following including training. You can use latex to represent it better. By the way where do you want to use this ?</p>

username: aaditya
cooked: <p><a class="mention" href="/u/balamurali_m">@Balamurali_M</a> Is it correct? What else I can add here?</p>
<p>I am writing for my thesis.</p>