KERAS TO pytorch model conversion

aneeka_azmat: <p>Hello</p>
<p><strong>keras code is given below:</strong></p>
<p>from tensorflow import keras<br>
from tensorflow.keras import layers</p>
<p>def transformer_encoder(inputs, head_size, num_heads, ff_dim, dropout=0):<br>
# Normalization and Attention<br>
x = layers.LayerNormalization(epsilon=1e-6)(inputs)<br>
x = layers.MultiHeadAttention(<br>
key_dim=head_size, num_heads=num_heads, dropout=dropout<br>
)(x, x)<br>
x = layers.Dropout(dropout)(x)<br>
res = x + inputs</p>
<pre><code># Feed Forward Part
x = layers.LayerNormalization(epsilon=1e-6)(res)
x = layers.Conv1D(filters=ff_dim, kernel_size=1, activation="relu")(x)
x = layers.Dropout(dropout)(x)
x = layers.Conv1D(filters=inputs.shape[-1], kernel_size=1)(x)
return x + res
</code></pre>
<p>def build_model(<br>
input_shape,<br>
head_size,<br>
num_heads,<br>
ff_dim,<br>
num_transformer_blocks,<br>
mlp_units,<br>
dropout=0,<br>
mlp_dropout=0,<br>
):<br>
inputs = keras.Input(shape=input_shape)<br>
x = inputs<br>
for _ in range(num_transformer_blocks):<br>
x = transformer_encoder(x, head_size, num_heads, ff_dim, dropout)</p>
<pre><code>x = layers.GlobalAveragePooling1D(data_format="channels_first")(x)
for dim in mlp_units:
    x = layers.Dense(dim, activation="relu")(x)
    x = layers.Dropout(mlp_dropout)(x)
outputs = layers.Dense(n_classes, activation="softmax")(x)
return keras.Model(inputs, outputs)
</code></pre>
<p>input_shape = x_train.shape[1:]</p>
<p>model = build_model(<br>
input_shape,<br>
head_size=256,<br>
num_heads=4,<br>
ff_dim=4,<br>
num_transformer_blocks=4,<br>
mlp_units=[128],<br>
mlp_dropout=0.4,<br>
dropout=0.25,<br>
)</p>
<p>model.compile(<br>
loss=“sparse_categorical_crossentropy”,<br>
optimizer=keras.optimizers.Adam(learning_rate=0.0001),<br>
metrics=[“accuracy”],<br>
)</p>
<p>checkpoint_filepath = “transformer/best_model1.hdf5”<br>
print(os.listdir(“transformer”))</p>
<p>model.summary()<br>
model.load_weights(checkpoint_filepath)<br>
callbacks = [keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)]<br>
model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_filepath, monitor=‘loss’, verbose=1,save_best_only=True, mode=‘auto’, save_freq=‘epoch’,period=1)</p>
<h1>
<a name="model_checkpoint_callback-tfkerascallbacksmodelcheckpointfilepathmodelepoch02d-val_loss2fh5-1" class="anchor" href="#model_checkpoint_callback-tfkerascallbacksmodelcheckpointfilepathmodelepoch02d-val_loss2fh5-1"></a>model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(filepath=‘model.{epoch:02d}-{val_loss:.2f}.h5’)</h1>
<h1>
<a name="model_checkpoint_tsboard-tfkerascallbackstensorboardlog_dirtransformer-2" class="anchor" href="#model_checkpoint_tsboard-tfkerascallbackstensorboardlog_dirtransformer-2"></a>model_checkpoint_tsboard = tf.keras.callbacks.TensorBoard(log_dir=‘./transformer’)</h1>
<p>model.fit(<br>
x_train,<br>
y_train,<br>
validation_split=0.2,<br>
epochs=85,<br>
batch_size=1,<br>
callbacks=[callbacks,model_checkpoint_callback]<br>
)</p>
<p><strong>I have tried to convert into pytorch given below:</strong></p>
<p>import torch<br>
import torch.nn as nn</p>
<p>def transformer_encoder(inputs, head_size, num_heads, ff_dim, dropout=0):<br>
# Normalization and Attention<br>
x = nn.LayerNorm(epsilon=1e-6)(inputs)<br>
x = nn.MultiheadAttention(<br>
key_dim=head_size, num_heads=num_heads, dropout=dropout<br>
)(x, x)<br>
x = nn.Dropout(dropout)(x)<br>
res = x + inputs</p>
<pre><code># Feed Forward Part
x = nn.LayerNormalization(epsilon=1e-6)(res)
x = nn.Conv1D(filters=ff_dim, kernel_size=1, activation="relu")(x)
x = nn.Dropout(dropout)(x)
x = nn.Conv1D(filters=inputs.shape[-1], kernel_size=1)(x)
return x + res
</code></pre>
<p>def build_model(<br>
input_shape,<br>
head_size,<br>
num_heads,<br>
ff_dim,<br>
num_transformer_blocks,<br>
mlp_units,<br>
dropout=0,<br>
mlp_dropout=0,<br>
):<br>
inputs = torch.tensor(shape=input_shape)<br>
x = inputs<br>
for _ in range(num_transformer_blocks):<br>
x = transformer_encoder(x, head_size, num_heads, ff_dim, dropout)</p>
<pre><code>x = nn.AvgPool1d(data_format="channels_first")(x)
for dim in mlp_units:
    x = nn.linear(dim, activation="relu")(x)
    x = nn.Dropout(mlp_dropout)(x)
outputs = nn.linear(n_classes, activation="softmax")(x)
return keras.Model(inputs, outputs)
</code></pre>
<p>Q1: what is should write at the place of return keras.Model(inputs, outputs)?</p>
<p>Q2: I want to pass data using dataloader pytorch as:<br>
for batch_idx, (images_mls, labels_mls, labels_sec_mls) in enumerate(trainloader_mls):<br>
will this conversion work for it?</p>
<p>Thank you</p>

ptrblck: <ol>
<li>
<p>You could define a custom <code>nn.Module</code> as described e.g. <a href="https://pytorch.org/tutorials/beginner/examples_nn/polynomial_module.html">here</a>, initialize all layers in the <code>__init__</code> and define the forward pass in the <code>forward</code> method.</p>
</li>
<li>
<p>Yes, a <code>DataLoader</code> would work as it will return batches of samples (usually data and targets) and you can pass the data directly to the model.</p>
</li>
</ol>

aneeka_azmat: <p>Thanks for your reply, I have tried to convert but:</p>
<ol>
<li>what should I write instead of inputs = keras.Input(shape=input_shape)?</li>
<li>what is replacement for return keras.Model(inputs, outputs) in pytorch?</li>
</ol>
<p>please check my converted code below, if there is any mistake , make me correct please. Thanks:</p>
<p>class transformer_encoder(nn.Module):<br>
def <strong>init</strong>(self,inputs, head_size, num_heads, ff_dim, dropout=0):<br>
Super(transformer_encoder, self).<strong>init</strong>()<br>
self.x = nn.LayerNorm(eplison = 1e-6)(inputs)<br>
self.x = nn.MultiHeadAttention(key_dim = head_size, num_heads = num_heads, dropout=dropout)(self.x, self.x)<br>
self.x = nn.Dropout(droput)(self.x)<br>
res = self.x + inputs<br>
def forward(self, x):<br>
self.x = nn.LayerNorm(eplison = 1e-6)(res)<br>
self.x = nn.conv1D(filters=ff_dim, kernel_size = 1, activation=‘relu’)(self.x)<br>
self.x = nn.Dropout(dropout)(self.x)<br>
self.x = nn.Con1D(filters=inputs.shape[-1], kernel=1)(self.x)<br>
return self.x+ res</p>
<p>class build_model(nn.Module):<br>
def <strong>init</strong>(self, input_shape,head_size,num_heads,ff_dim,num_transformer_blocks,mlp_units,dropout=0,mlp_dropout=0,transformer_encoder):<br>
super(build_model, self).<strong>init</strong>()<br>
inputs = keras.Input(shape=input_shape)<br>
x = inputs<br>
for _ in range(num_transformer_blocks):<br>
x = transformer_encoder(x, head_size, num_heads, ff_dim, dropout)<br>
x = nn.AvgPool1d(data_format=“channels_first”)(x)<br>
for dim in mlp_units:<br>
x = nn.Linear(dim, activation=“relu”)(x)<br>
x = nn.Dropout(mlp_dropout)(x)<br>
outputs = nn.Linear(n_classes, activation=“softmax”)(x)<br>
return keras.Model(inputs, outputs)</p>

Anshumaan_Dash: <p>You are defining the layers twice in your torch model class. You declare them in <code>__init__</code> method and use them as member variables as <code>self.something</code> within the <code>forward</code> method. You can refer to this: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html" class="inline-onebox" rel="noopener nofollow ugc">Module — PyTorch 1.11.0 documentation</a></p>

aneeka_azmat: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e67540e7059dd3580e372dd3d7a3a08bca52e217.png" data-download-href="https://discuss.pytorch.org/uploads/default/e67540e7059dd3580e372dd3d7a3a08bca52e217" title="t1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e67540e7059dd3580e372dd3d7a3a08bca52e217_2_690x272.png" alt="t1" data-base62-sha1="wSJ21MCZAnLQAOvSMsPvCsmZWdh" width="690" height="272" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e67540e7059dd3580e372dd3d7a3a08bca52e217_2_690x272.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e67540e7059dd3580e372dd3d7a3a08bca52e217_2_1035x408.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e67540e7059dd3580e372dd3d7a3a08bca52e217_2_1380x544.png 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">t1</span><span class="informations">3824×1512 141 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
can you please find mistakes and tell me changes?</p>

Anshumaan_Dash: <p>It’s like I mentioned in the previous comment, your <code>__init__</code> and <code>forward</code> methods are all wrong. The <code>__init__</code> method is used to build the layers → it doesn’t accept inputs, nor does it return anything. The <code>forward</code> method accepts only the inputs and defines the flow of data using the layers you defined in the <code>__init__</code> method. The syntax you are using is from Keras’s functional API. Please have a look at the reference doc I shared earlier. It should be easy enough.</p>

aneeka_azmat: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f51ecaf65634ba5ac037925b9f21fc07f1f30a62.png" data-download-href="https://discuss.pytorch.org/uploads/default/f51ecaf65634ba5ac037925b9f21fc07f1f30a62" title="t2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f51ecaf65634ba5ac037925b9f21fc07f1f30a62_2_690x272.png" alt="t2" data-base62-sha1="yYqX7xIl42hRBygMfzY3EoljFDk" width="690" height="272" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f51ecaf65634ba5ac037925b9f21fc07f1f30a62_2_690x272.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f51ecaf65634ba5ac037925b9f21fc07f1f30a62_2_1035x408.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f51ecaf65634ba5ac037925b9f21fc07f1f30a62_2_1380x544.png 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">t2</span><span class="informations">3824×1512 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
is it right now?</p>

aneeka_azmat: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c3d488396351cac701c5159037af4a64a5660ad.png" data-download-href="https://discuss.pytorch.org/uploads/default/2c3d488396351cac701c5159037af4a64a5660ad" title="t3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c3d488396351cac701c5159037af4a64a5660ad_2_690x272.png" alt="t3" data-base62-sha1="6jmj3VHCi0bLGEoYkgyMKJIdK33" width="690" height="272" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c3d488396351cac701c5159037af4a64a5660ad_2_690x272.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c3d488396351cac701c5159037af4a64a5660ad_2_1035x408.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c3d488396351cac701c5159037af4a64a5660ad_2_1380x544.png 2x" data-dominant-color="FDFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">t3</span><span class="informations">3824×1512 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I am unable to finalize second function <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>