Activation Function: Absolute Function,One Function Behaves more Individualized
Authors: Jinxin Wei
Zhe Hou
Venue: Submitted to ICLR 2023 (2023)
Keywords: activation function
absolute function
individualization
universality
over-fitting
Z-Score
abstract network
concrete network，stimulation

Abstract: Inspire by nature world mode, a activation function is proposed. It is absolute function.Through test on mnist dataset and fully-connected neural network and convolutional neural network, some conclusions are put forward. The line of accuracy of absolute function is a little shaken that is different from the line of accuracy of relu and leaky relu. The absolute function can keep the negative parts as equal as the positive parts, so the individualization is more active than relu and leaky relu function. The absolute function is less likely to be over-fitting.  Through test on mnist and autoencoder, It is that the leaky relu function can do classification task well, while the absolute function can do generation task well. Because the classification task need more universality and generation task need more individualization. The pleasure irritation and painful irritation is not only the magnitude differences, but also the sign differences, so the negative parts should keep as a part. Stimulation which happens frequently is low value, it is showed around zero in figure 1 . Stimulation which happens accidentally is high value, it is showed far away from zero in figure 1. So the high value is the big stimulation, which is individualization.

Activation Function: Absolute Function,One Function Behaves
more Individualized
Abstract: Inspire by nature world mode, activation function is proposed. is
absolute function.Through test mnist dataset and fully-connected neural network and convolutional neural network, some conclusions are put forward. The line of accuracy of absolute function a little shaken that is different from the line of accuracy of relu and leaky relu. The absolute function can keep the negative parts as equal the positive parts, the individualiizaion is more active than relu and leaky relu function. In order to generalization, the individualization is the reason of shake, the accuracy may be good in some set and may be worse in some set. The absolute function less likely to be over-fitting, The batch size small, the individualization is clear, vice versa. Through test mnist and autoencoder, It is that the leaky relu function can do classification task well, while the absolute function can do generation task well. Because the classification task need more universality and generation task need more individuullzzaioon The pleasure irritation and painful irritation is not only the magnitude differences, but also the sign differences, the negative part should keep as a part. Stimulation which happens frequently is low value, showed around zero in figure 1 Stimulation which happens accidentally is high value, is showed far away from zero in figure 1. So the high value is the big stimulation, which is individualization.
Keyword: activation function, absolute function, individualization, universality, over-fitting, abstract network, concrete network, stimulation
1.1. Preface
There are many activation function, such as sigmoid, tanh, relu, leaky relu, elu. The function frequently used relu[1]. The relu function has dead relu problem that is some neuron never be activated, the related parameter never be updated. The absolute function doesn' have dead relu problem, and gradient disappearance and gradient explosion, however has some interesting characteristics.
The image of absolute function showed below. The formula is y= /=|x|.

---

Fig1:The Image of Absolute Function
2.1. The test of absolute function on fully-connected neural network
First, test the activation function which include relu leaky relu and absolute mnist dataset and fully-connected neural network. The network is layers. The first 5000 samples are validation set, the left are training set(7]. Optimizer is adam[6), loss is SpareeCategricalrrosennroyy metric accuracy|2). The following picture is the test result.
From the figure the accuracy of absolute function in validation set is lower than the accuracy of relu and leaky relu in validation set when epochs are small. But when the epochs grow, the accuracy of absolute function in validation set is almost equal with the accuracy of relu and leaky relu in validation set. The line of accuracy of absolute function is little shaken that is different from the line of accuracy of relu and leaky relu. The loss of absolution function is almost equal with the loss of relu and leaky relu. Because the absolute function keep the negative parts as equal as the positive parts, the individualization is more active than relu and leaky relu function. Why? Let's change the batch size, the results are in figure 3.

---

Training and validation accuracy
1.000
0.975
0.950
0.925
accuracy
0.900
relu Training relu Validation acc
0.875
leakyrelu Training acc
0.850
leakyrelu Validation acc abs Training acc
0.825
abs Validation acc
0.800
5
10
15
25
Epochs
Fig2.1:Training and Validation Accuracy, size is 32
Training and validation loss
2.00
relu Training loss
1.75
relu Validation loss
leakyrelu Training loss
150
leakyrelu Validation loss abs Training loss
125
abs Validation loss
Loss
100
0.75
0.50
0.25
0.00
5
10
15
20
25
30
Epochs
Fig2.2:Training and Validation Basch Size is 32
In figure 3, batch size 64, the line no longer shaken like before, it behaves like the line of relu and leaky relu. The line of relu and leaky relu function are more universality and stable than the line of absolute function. It filters the negative parts, so it loses the individualization partly. In order to generalization, the individualiiation the reason of shake, the accuracy may be good in some set and may be worse in some set. The absolute function is less likely to be over-fitting.

---

Training and validation accuracy
1.000
0.975
0.950
0.925
accuracy
0.900
relu Training
0.875
relu Validation acc
leakyrelu Training acc
0.850
leakyrelu Validation acc abs Training acc
0.825
abs Validation acc
0.800
10
20
50
Epochs
Fig3:Training and Validation Accuracy, Batch Size is 64
2.2. The test of absolute function convolutional neural network
Let's change the neural network, the result convolutional neural network and mnist dataset is showed in figure 4. The convolutional neural network has 3 layers convolutional neural network and 2 layers fully connected network. The shake phenomenon is more clear than on fully-connected neural network when batch size is 32. When batch size increase, the shake phenomenon is more weaker. It can be anti-over-fitting when batch_siize is small. The loss which decrease firstly but increase then is odd when batch size is 32, however the loss is stable like relu and leaky relu when batch size is 128. Correspondingly, the accuracy is increase step by step. In order to increase the accuracy, the model has to learn the individualization which increases the loss. The relu and leaky relu function can not learn the individualization like the absolute function, so its loss is stable. The batch size is small, the individuaaliatiin is clear, vice versa. If you want to change the individualization of absolute function, just change the batch size.
Training and validation accuracy
1.00
0.98
0.96
accuracy
0.94
relu Training acc
relu Validation acc
leakyrelu Training acc
0.92
leakyrelu Validation acc abs Training acc
abs Validation acc
0.90
10
20
30
60
70
Epochs

---

Tig4.n:Training and Validation Accuracy,Batch Size is 32
Training and validation loss
2000
relu Training loss
175
relu Validation loss
leakyrelu Training loss
150
leakyrelu Validation loss abs Training loss
125
abs Validation loss
Loss
1.00
0.75
0.50
0.25
0.00
30
40
50
Epochs
Fig4.2:Training and Validation Loss, Batch Size is 32
Training and validation accuracy
1.00
0.98
accuracy
0.96
0.94
relu Training acc
relu Validation acc
leakyrelu Training acc
0.92
leakyrelu Validation acc abs Training acc
abs Validation acc
0.90
0
10
20
30
40
50
60
70
Epochs
Fig4.3:Training and Validation Accuracy, Batch Size is 64
Training and validation loss
2.00
relu Training loss
175
relu Validation loss
leakyrelu Training
150
leakyrelu Validation loss
Training
125
Validation
Loss
1.00
0.75
0.50
0.25
0.00
30
40
50
Epochs
Fig4.4 Tiga.::Training and Validation Loss, Batch Size is 64

---

Training and validation accuracy
1.00
0.98
accuracy
0.96
0.94
relu Training acc
relu Validation acc
leakyrelu Training acc
0.92
leakyrelu Validation acc abs Training acc
abs Validation acc
0.90
0
10
20
30
40
50
60
70
Epochs
Figai5: and Validation Accuracy, Batch Size is 128
Training and validation loss
2.00
relu Training loss
175
relu Validation loss
leakyrelu Training
1.50
leakyrelu Validation loss
Training
125
Validation
Loss
1.00
0.75
0.50
0.25
0.00
30
40
50
Epochs
G:raannn and Validation Loss, Batch Size is 128
In order to visualize the intermediate activation, one more test is conducted. The convolutional neural network has 3 layers convolutional neural network and layers fully connected network. The test result is showed in Fig4.7 and Fig 4.8. In Fig 4.7,the intermediate activation is sparse and blur with relu activation; Oppositely in Fig 4.8, the intermediate activation dense and clear with absolute function. Because the relu activation function filter the negative part and the absolute activation function keep the negative part. Because the feature the absolute function shows is clear the generation work can be done by absolute function.
7
Fig4.7.1 Visualize Intermediate Activation with Relu Activation(First Convolution Layer)

---

35
120
149
350
Fig4.7.2 Visualize Intermediate Activation with Relu activation (Second Convolution Layer)
F.g4.7.3 Visualize Intermediate Activation with Relu activation (Third Convolution
Layer)
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
300
150
200
250
300
X5
Fig4.8.1 Visualize Intermediate Activation with Absolute activation (First Convolution Layer)
Fig4.8.2 Visualize Intermediate Activation with Absolute activation (Second Convolution Layer)

---

F.34.8.3 Visualize Intermediate Activation with Absolute activation (Third Convolution Layer)
3. The test of absolute function on autoencoder generation
Because the absolute function has more individualization, so I choose it to do autoencoder's generation[3] work.
Abstract network(namely prediction network, always used as classification and regression) is common now. But the concrete network(namely generation network) which generates concrete information from concept or label rare. Its principle is showed in figure 5. The test is on mnist dataset and convolutional neural network(5). The convolutional neural network(abstract network) is like LeNet, it has 4 layers convolutional neural network and 2 layers fully connected network. The concrete network is the inverse function of abstract network, has 6 layers. Optimizer is adam, loss is mse. You can read my paper[3] for the detail. The test result is showed in figure 6. In figure 6, the left image is input, the label is argmax function of the predict of output of abstract network, the right image is the output of concrete network.
Concretennetwork:geeerrre
Abstrachetworkprediirrreeggiieee
Inverserofy=wxxttbb.
Inverse-ofy=wwx:+bb
Leaky-Relu-
sigmoid.
X1
Inverse singmoi..
Swerss--f-leekkyRRlul
Fig5 The Functionally Separate Auto-encoder

---

2
4
2 4
0
4
5
6
9
0
6
9
5
9
7
4
5
3
9
6
5 5
9
6
6
4
Fig6.1 Leaky Relu Function,Epoch=1
In figure 6.1, when epoch=1, the labels of input are all right. But the image of output is very blurred. Meanwhile, in figure 6.2, the labels of input are not all right. But the image of output clear. One conclusion can easily draw. It is that the leaky relu function can do classification task well, while the absolute function can do generation task well. Because the classification task need more universality and generation task need more individualization. In figure 6.2 and 6.3, epoch=30, the result almost the same between the two activation function. One different is the background color of absolute function is whiter than leaky relu function. So the absolute function can do generation task well.
7
7
2
2
4
4
2
1
0
4
4
4
6
9
S
9
9
9
5
D
O
6
6
9
9
O
O
0
9
0
5
5
9 9
9
7
7
3
4
4
5
5
4
9
9
6
6
4
6
5 5
S
4
4
9
5
Fig6.2 2 Absolute Function, 1pochh0

---

7
7
2
2
/
4
4
7
2
1
0
4
4
4
9
S
6
9
5
D
O
6
6
9
9
O
0
6
9
0
1
5
5
9
9
7
7
3
4
4
5
9
3
9
9
6
6
6
5 5
5
4
4
9
6
6
4
Fig6.3 Leaky Relu Function, Epoch=30
7
7
2
2
4
4
2
1
0
4
4
4
6
9
S
C
9
9
4
9
5
D
O
6
6
9
9
0
O
0
6
9
0
1
5
5
9
9
7
7
3
4
4
9
9
6
6
6
6
5
5
4 4
4
7
6
6
5
F Absolute Junction,Epoch-3]
4.Some Thinking
Should we keep the negative parts of the inputs? think we can learn from other fields. The color has white and black, the temperature has hot and cool, the smell has fragrant and smelly, the taste has bitter and sweet. All information our body received are opposite mode. The pleasure irritation and painful irritation not only the magnitude differences, but also the sign differences, so the negative parts should keep as part. According to probability principle, nature word is normal distribution(4]. Stimulation which happens frequently is low value, it is showed around zero in figure 1 Stimulation which happens accidentally is high value, it is showed far away from zero figure So the high value the big stimulation, which is individualization.
References

---

1. lan Goodfellow, Yoshua Bengio, Aaron Courville, "Deep Learning", MIT Press, 2016
2. Tensorflow Tutorials and Apis, https:/tensorflow. wwwooll.cn/learr last accessed 2021/12/20
3..Jin-xin Wei, Qun-ying Ren, A Functionally Separate Autoencoder, Future Technology Conference, 2020/10
4.Sheldon M.Ross, "A First Course in Probability" "aarbiiitt""PPersooo" Pearson Prentice Hall, 2009/01/07
5.Aurélien Géron, "Hands-On Machine Learning with Scikit-Learn and TensorFlow", O'Reilly Media, 2017/01/15
Frangoois Chollet, "Deep Learning with Python", Manning Publications, 2017/10
7.Tom Mitchell, "Machine Learning", McGraw Hill Science, 2008/03


Reviews:
[{"review_id": "wGynI2ERCVP", "reviewer": "Reviewer_z5g5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "N/A", "summary_of_the_review": "This submission is not anonymous, the authors didn't use submission template either. ", "strength_and_weaknesses": "N/A", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "QloZ8UNifPo", "reviewer": "Reviewer_FJEP", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is an invalid submission that is non-anonymous and informal-formatted. Thus, it should be desk rejected. ", "summary_of_the_review": "Desk Reject.", "strength_and_weaknesses": "NA", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "J9kOEfKDND", "reviewer": "Reviewer_p6j5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " I treat this paper as desk rejected.", "summary_of_the_review": "Desk rejected.", "strength_and_weaknesses": "Desk rejected.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "E1ZiAxucX4", "reviewer": "Reviewer_YHRv", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I believe this article should be rejected since it does not follow ICLR's submission requirements (format, double-blind, etc.).", "summary_of_the_review": "N/A", "strength_and_weaknesses": "N/A", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject