激活函数

在计算网络中， 一个节点的激活函数定义了该节点在给定的输入或输入的集合下的输出。标准的计算机芯片电路可以看作是根据输入得到开（1）或关（0）输出的數位電路激活函数。这与神经网络中的线性感知机的行为类似。然而，只有非線性激活函数才允許這種網絡僅使用少量節點來計算非平凡問題。 在人工神經網絡中，這個功能也被稱為傳遞函數。 说明 ^ 若一函数是连续的，则称其为 C 0 {\displaystyle C^{0}} 函数；若一函数 n {\displaystyle n} 阶可导，并且其 n {\displaystyle n} 阶导函数连续，则为 C n {\displaystyle C^{n}} 函数（ n ≥ 1 {\displaystyle n\geq 1} ）；若一函数对于所有 n {\displaystyle n} 都属于 C n {\displaystyle C^{n}} 函数，则称其为 C ∞ {\displaystyle C^{\infty }} 函数，也称光滑函数。 ^ 此處H是單位階躍函數。 ^ α是在訓練時間從均勻分佈中抽取的隨機變量，並且在測試時間固定為分佈的期望值。 ^ ^ ^ 此處 σ {\displaystyle \sigma } 是邏輯函數。 说明 ^ 此處δ是克羅內克δ函數。 邏輯函數 線性整流函數 Softmax函數 人工神經網路 深度學習 Bergstra, James; Desjardins, Guillaume; Lamblin, Pascal; Bengio, Yoshua. Quadratic polynomials learn better image features". Technical Report 1337. Département d’Informatique et de Recherche Opérationnelle, Université de Montréal. 2009. （原始内容存档于2018-09-25）.  Glorot, Xavier; Bengio, Yoshua, Understanding the difficulty of training deep feedforward neural networks (PDF), International Conference on Artificial Intelligence and Statistics (AISTATS’10), Society for Artificial Intelligence and Statistics, 2010, （原始内容存档 (PDF)于2017-04-01）  Carlile, Brad; Delamarter, Guy; Kinney, Paul; Marti, Akiko; Whitney, Brian. Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs). 2017-11-09. arXiv:1710.09967  [cs.LG].  He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. 2015-02-06. arXiv:1502.01852  [cs.CV].  Xu, Bing; Wang, Naiyan; Chen, Tianqi; Li, Mu. Empirical Evaluation of Rectified Activations in Convolutional Network. 2015-05-04. arXiv:1505.00853  [cs.LG].  Clevert, Djork-Arné; Unterthiner, Thomas; Hochreiter, Sepp. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). 2015-11-23. arXiv:1511.07289  [cs.LG].  Klambauer, Günter; Unterthiner, Thomas; Mayr, Andreas; Hochreiter, Sepp. Self-Normalizing Neural Networks. 2017-06-08. arXiv:1706.02515  [cs.LG].  Jin, Xiaojie; Xu, Chunyan; Feng, Jiashi; Wei, Yunchao; Xiong, Junjun; Yan, Shuicheng. Deep Learning with S-shaped Rectified Linear Activation Units. 2015-12-22. arXiv:1512.07030  [cs.CV].  Forest Agostinelli; Matthew Hoffman; Peter Sadowski; Pierre Baldi. Learning Activation Functions to Improve Deep Neural Networks. 21 Dec 2014. arXiv:1412.6830  [cs.NE].  Glorot, Xavier; Bordes, Antoine; Bengio, Yoshua. Deep sparse rectifier neural networks (PDF). International Conference on Artificial Intelligence and Statistics. 2011. （原始内容存档 (PDF)于2018-06-19）.  Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning. [2018-06-13]. （原始内容存档于2018-06-13）.  Searching for Activation Functions. [2018-06-13]. （原始内容存档于2018-06-13）.  Godfrey, Luke B.; Gashler, Michael S. A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks. 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management: KDIR. 2016-02-03, 1602: 481–486. Bibcode:2016arXiv160201321G. arXiv:1602.01321 .  Goodfellow, Ian J.; Warde-Farley, David; Mirza, Mehdi; Courville, Aaron; Bengio, Yoshua. Maxout Networks. JMLR WCP. 2013-02-18, 28 (3): 1319–1327. Bibcode:2013arXiv1302.4389G. arXiv:1302.4389 .