PyTorch の活性化関数をグラフ化した

#はじめに

PyTorch の パッケージ [TORCH.NN.FUNCTIONAL](TORCH.NN.FUNCTIONAL) の[非線形活性化関数](https://pytorch.org/docs/stable/nn.functional.html#non-linear-activation-functions) （Non-linear activation functions）をグラフ化しました。

#目次

# TORCH.NN.FUNCTIONAL 活性化関数のグラフ化

各関数のグラフをを一覧にしました。（左側の青いグラフ）
右側に微分値もあわせてグラフ化してみました。（右側のオレンジのグラフ）

##relu

```Python
import torch
import torch.nn.functional as F
import matplotlib.pyplot as plt

'''
#############################################
## グラフを描画する
#############################################
'''
def drawGraph(x, y, y_dash):
  fig = plt.figure(figsize=(12, 4))
  ax1 = fig.add_subplot(1, 2, 1)
  ax2 = fig.add_subplot(1, 2, 2)
  ax1.plot(x, y)
  ax2.scatter(x, y_dash, s=1, color='orange')
  plt.show()

x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.relu(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![relu.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/f8f8756a-6aff-eb2d-b361-b45a57412de5.png)


##hardtanh

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.hardtanh(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![hardtanh.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/a211164f-859e-72bd-3654-5b974e1785a1.png)


##relu6

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.relu6(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![relu6.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/6d926015-57da-7145-bab8-1e4d29c2d474.png)


##elu

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.elu(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![elu.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/34a40ba4-275c-7591-16c4-4f264d543ce2.png)


##selu

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.selu(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![selu.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/3d79c7ac-ac31-553a-35ef-06473dd3e328.png)


##celu

```Python
x=torch.linspace(-1, 1, 1000, dtype=torch.float, requires_grad=True)
y=F.celu(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![celu.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/2ef26ba4-08d4-c43c-6efb-3ff5de745c8b.png)



##leaky_relu

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.leaky_relu(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![leaky_relu.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/94bf6ecc-b5ad-65b7-3a2c-f60182158ee6.png)


##prelu

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.prelu(x, torch.tensor([0.25]))
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![prelu.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/6d86039f-16b6-6209-b451-d45cd11c3e28.png)


##rrelu

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.rrelu(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![rrelu.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/98a56759-1187-3002-776f-92d8b70d5938.png)


##glu

[PyTorch の公式ドキュメント](https://pytorch.org/docs/stable/nn.functional.html#glu) の記載を見ると、  glu は入力を2つに分け、
glu(x1,x2) = x1 * sigmoid(x2)
で計算されます。

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float)
y=F.glu(x)

# xを2つに分解する
x1=torch.linspace(-6, 0, 500, dtype=torch.float, requires_grad=True)
x2=torch.linspace(0, 6, 500, dtype=torch.float, requires_grad=True)
y=torch.mul(x1, torch.sigmoid(x2))
z=y.sum()
z.backward()
#x1 x2 で自動微分
z_dash1=x1.grad
z_dash2=x2.grad
z_dash3=torch.add(z_dash1, z_dash2)
drawGraph(x2.detach().numpy(), y.detach().numpy(), z_dash3.detach().numpy())
```

![glu.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/5c838a59-9d2c-6133-ec48-0903dcfac8cc.png)


##gelu

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.gelu(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![gelu.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/10a03c16-7e1a-86a6-c90d-0b73b4ab0119.png)


##logsigmoid

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.logsigmoid(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![logsigmoid.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/b360f553-1312-38aa-ef62-6334d78a19a7.png)


##hardshrink

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.hardshrink(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![hardshrink.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/dacd147e-60c2-38da-b0b7-7eef55aedb57.png)


##tanhshrink

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.tanhshrink(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![tanhshrink.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/1f2eabf7-7ca6-1790-f1d4-5fb231dfedc0.png)


##softsign

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.softsign(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![softsign.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/8aea0e02-63ec-4ae5-2dc9-d0b6eac4f3ba.png)


##softplus

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.softplus(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![softplus.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/e298d124-e5bf-becd-0240-79533076c87b.png)


##softmin

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.softmin(x,  0)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![softmin.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/8753e4f3-307e-efd8-519f-d7690d1d9a9a.png)


##softmax

```Python
x=torch.linspace(-6, 6, 200, dtype=torch.float)
y=F.softmax(x)
plt.plot(x.numpy(), y.numpy())
plt.show()

UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
```

![softmax.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/ea92ce7c-c2ef-772d-1d42-3af5a8c70ca6.png)

ソフトマックスは2次元だとうまくグラフ化できていないような気がします。
機会があればもう少し調べてみたいと思います。



##softshrink

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.softshrink(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![softshrink.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/b83c5fbb-8d86-5016-33ae-1c966492d152.png)


##gumbel_softmax

```Python
x=torch.linspace(-6, 6, 200, dtype=torch.float)
y=F.gumbel_softmax(x)
plt.plot(x.numpy(), y.numpy())
plt.show()
```

![gumbel_softmax.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/5894b9f1-cf22-8166-e3ea-6a33815fc5ca.png)

gumbel_softmax もソフトマックスと同じくうまくグラフ化できていないようです。


##log_softmax

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.log_softmax(x, 0)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())
```

![log_softmax.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/eea9a569-7ad5-6564-4d20-342110193cff.png)


##tanh

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.tanh(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())

UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
```

![tanh.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/7f705de9-48d1-052e-728f-6aa5a6296de6.png)


##sigmoid

```Python
x=torch.linspace(-6, 6, 1000, dtype=torch.float, requires_grad=True)
y=F.sigmoid(x)
z=y.sum()
z.backward()
z_dash=x.grad
drawGraph(x.detach().numpy(), y.detach().numpy(), z_dash.detach().numpy())

UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
```

![sigmoid.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/545343/8c7251cf-92cf-ea34-cdbe-c191dd186649.png)




#履歴
2020/04/18 初版公開