DQN always gives same output regardless of input

Jaelee: <p>I am solving combinational optimization problem with DQN.<br>
My goal is to reach the optimal state</p>
<p>I just revised a little from the following pytorch DQN tutorials</p>
<p><a href="https://tutorials.pytorch.kr/intermediate/reinforcement_q_learning.html" class="onebox" target="_blank" rel="nofollow noopener">https://tutorials.pytorch.kr/intermediate/reinforcement_q_learning.html</a></p>
<pre><code class="lang-auto">
class DQN(nn.Module):

    def __init__(self, h, w, outputs):
        super(DQN, self).__init__()
        self.channel = 7
        self.conv1 = nn.Conv2d(self.channel, 16, kernel_size=3, stride=2)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, stride=2)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=1, stride=2)
        self.bn3 = nn.BatchNorm2d(32)


        def conv2d_size_out(size, kernel_size=5, stride=2):
            return (size - (kernel_size - 1) - 1) // stride + 1

        convw = 2
        convh = 2
        linear_input_size = convw * convh * 32
        self.head = nn.Linear(linear_input_size, outputs)


    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))

</code></pre>
<pre><code class="lang-auto">def select_action(state):
    global steps_done
    sample = random.random()
    #sample = 1.0
    eps_threshold = EPS_END + (EPS_START - EPS_END) * \
        math.exp(-1. * steps_done / EPS_DECAY)
    steps_done += 1
    if sample &gt; eps_threshold:
        with torch.no_grad():
            # t.max (1)은 각 행의 가장 큰 열 값을 반환합니다.
            # 최대 결과의 두번째 열은 최대 요소의 주소값이므로,
            # 기대 보상이 더 큰 행동을 선택할 수 있습니다.
            return policy_net(state).max(1)[1].view(1, 1)
    else:
        return torch.tensor([[random.randrange(n_actions)]], device=device, dtype=torch.long)


def optimize_model():
    if len(memory) &lt; BATCH_SIZE:
        return
    transitions = memory.sample(BATCH_SIZE)

    batch = Transition(*zip(*transitions))

    non_final_mask = torch.tensor(tuple(map(lambda s: s is not None,
                                          batch.next_state)), device=device, dtype=torch.bool)

    non_final_next_states = torch.cat([s for s in batch.next_state
                                                if s is not None])
    state_batch = torch.cat(batch.state)
    action_batch = torch.cat(batch.action)
    reward_batch = torch.cat(batch.reward)


    state_action_values = policy_net(state_batch).gather(1, action_batch)  # double DQN
    
    next_state_values = torch.zeros(BATCH_SIZE, device=device, dtype=torch.double)
    next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0].detach()



    expected_state_action_values = (next_state_values * GAMMA) + reward_batch



    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

    optimizer.zero_grad()
    loss.backward()
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()

</code></pre>
<p>After some learning, all output gives same value regardless of input<br>
So, I changed the reward to 0 for every action, but the situation was same</p>
<p>Please give me an advice</p>

Artlind: <p>Did you try to reduce the learning rate ?</p>

Viktor_Todosijevic: <p>Hey, I’m having a similar issue. Did you manage to solve it?</p>

vmoens: <p>Check this answer</p><aside class="quote" data-post="3" data-topic="47615">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vmoens/48/26121_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/output-of-actor-is-almost-same-for-different-states/47615/3">Output of actor is (almost)same for different states</a> <a class="badge-category__wrapper " href="/c/reinforcement-learning/6"><span data-category-id="6" data-drop-close="true" class="badge-category " title="A section to discuss RL implementations, research, problems"><span class="badge-category__name">reinforcement-learning</span></span></a>
  </div>
  <blockquote>
    You’re using batch norm so it is expected that after initialization the output is roughly the same. 
If you use batch norm and apply an affine transform to your image, the result will be the same. 
Aside, using batch norm in RL is not always a great idea as you may loose some “temporal” information about your data. It is not a common thing to do. 
Check our <a href="https://github.com/pytorch/rl/blob/726dc425a99bead4e4a1c550d997c3ec9bdaf8c1/torchrl/modules/models/models.py#L259" rel="noopener nofollow ugc">ConvNet in torchrl</a> for a quick implementation
  </blockquote>
</aside>