title: ÐÑÐµÐ½ÐºÐ° ÑÑÑÐµÐºÑÐ¾Ð² Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð² ÑÐ²ÐµÑÑÐ¾ÑÐ½ÑÑ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÑÑ
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: Evaluating the effects of hyperparameters in convolutional neural networks
Creators: name: ÐÑÐ»Ð¾Ð½Ð´Ðµ, Ð¤Ð¸Ð»Ð¸Ð¿Ðµ
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2022
Subjects: ÐÐµÐ¹ÑÐ¾Ð½Ð½ÑÐµ ÑÐµÑÐ¸
Ð³Ð»ÑÐ±Ð¾ÐºÐ¾Ðµ Ð¾Ð±ÑÑÐµÐ½Ð¸Ðµ
Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÑ
deep learning
hyper-parameters

description: Ð¢ÐµÐ¼Ð° Ð²ÑÐ¿ÑÑÐºÐ½Ð¾Ð¹ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÑ: Â«ÐÑÐµÐ½ÐºÐ° ÑÑÑÐµÐºÑÐ¾Ð² Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð² ÑÐ²ÐµÑÑÐ¾ÑÐ½ÑÑ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÑÑÂ». ÐÐ°Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ ÑÑÑÐµÐºÑÐ¾Ð² Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÐµÐ¹ Ð² ÑÐ²ÐµÑÑÐ¾ÑÐ½ÑÑ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÑÑ Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ Ð½Ð°Ð±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ CIFAR-10. ÐÐ°Ð´Ð°ÑÐ¸, ÐºÐ¾ÑÐ¾ÑÑÐµ ÑÐµÑÐ°Ð»Ð¸ÑÑ Ð² ÑÐ¾Ð´Ðµ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ: ÐÐ±Ð·Ð¾Ñ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð¸ ÑÑÑÐµÑÑÐ²ÑÑÑÐ¸Ñ Ð¼ÐµÑÐ¾Ð´Ð¾Ð² Ð´Ð»Ñ Ð¿Ð¾Ð½Ð¸Ð¼Ð°Ð½Ð¸Ñ ÑÑÑÐµÐºÑÐ¾Ð² Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² ÑÐ²ÐµÑÑÐ¾ÑÐ½ÑÑ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÐµÐ¹. Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð±Ð°Ð·Ð¾Ð²Ð¾Ð¹ Ð¼Ð¾Ð´ÐµÐ»Ð¸ (ÑÐ²ÐµÑÑÐ¾ÑÐ½Ð¾Ð¹ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½Ð¾Ð¹ ÑÐµÑÐ¸) Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ Ð½Ð°Ð±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ CIFAR-10. ÐÐ·Ð°Ð¸Ð¼Ð¾Ð´ÐµÐ¹ÑÑÐ²Ð¸Ðµ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð² ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð½Ð¾Ð¹ ÑÐ²ÐµÑÑÐ¾ÑÐ½Ð¾Ð¹ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½Ð¾Ð¹ ÑÐµÑÐ¸. ÐÐ¸Ð·ÑÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ Ð¼Ð¾Ð´ÐµÐ»Ð¸ Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÑ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð². ÐÐ½Ð°Ð»Ð¸Ð· Ð²Ð·Ð°Ð¸Ð¼Ð¾Ð´ÐµÐ¹ÑÑÐ²Ð¸Ñ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð² ÑÐ²ÐµÑÑÐ¾ÑÐ½ÑÑ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÑÑ. ÐÐ¾Ð½Ð¸Ð¼Ð°Ð½Ð¸Ðµ Ð²Ð»Ð¸ÑÐ½Ð¸Ñ ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð°ÑÐ¸Ð¹ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² ÑÐ²ÐµÑÑÐ¾ÑÐ½ÑÑ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÐµÐ¹. Ð Ð°Ð±Ð¾ÑÐ° Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð° Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¾ÑÐºÑÑÑÐ¾Ð³Ð¾ Ð½Ð°Ð±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ CIFAR-10. ÐÐ° ÑÐ·ÑÐºÐµ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð¸ÑÐ¾Ð²Ð°Ð½Ð¸Ñ Python 3 Ñ Ð±Ð¸Ð±Ð»Ð¸Ð¾ÑÐµÐºÐ°Ð¼Ð¸ pytorch Ð¸ matplotlib. ÐÐµÐ¹ÑÐ¾Ð½Ð½ÑÐµ ÑÐµÑÐ¸ Ð¸Ð¼ÐµÑÑ Ð¼Ð½Ð¾Ð¶ÐµÑÑÐ²Ð¾ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð², Ð²ÐºÐ»ÑÑÐ°Ñ Ð²ÑÐ±Ð¾Ñ Ð¼ÐµÑÐ¾Ð´Ð° Ð³ÑÐ°Ð´Ð¸ÐµÐ½ÑÐ½Ð¾Ð³Ð¾ ÑÐ¿ÑÑÐºÐ°, ÑÑÐ½ÐºÑÐ¸Ð¸ Ð°ÐºÑÐ¸Ð²Ð°ÑÐ¸Ð¸, ÑÐµÐ³ÑÐ»ÑÑÐ¸Ð·Ð°ÑÐ¸Ñ, Ð½Ð¾ÑÐ¼Ð°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ, ÑÐ°Ð·Ð¼ÐµÑ Ð¿Ð°ÑÑÐ¸Ð¸ Ð¸ ÑÐ°Ðº Ð´Ð°Ð»ÐµÐµ. Ð¥Ð¾ÑÑ ÑÑÑÐµÑÑÐ²ÑÑÑ Ð¿ÐµÑÐµÐ´Ð¾Ð²ÑÐµ Ð¼ÐµÑÐ¾Ð´Ñ, Ð²Ð·Ð°Ð¸Ð¼Ð¾Ð´ÐµÐ¹ÑÑÐ²Ð¸Ðµ Ð¼ÐµÐ¶Ð´Ñ ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÐ¼Ð¸ Ð²Ð°ÑÐ¸Ð°Ð½ÑÐ°Ð¼Ð¸ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð¼Ð¾Ð¶ÐµÑ Ð±ÑÑÑ ÑÑÑÐ´Ð½Ð¾ Ð¿ÑÐµÐ´ÑÐºÐ°Ð·Ð°ÑÑ. ÐÐ»Ñ Ð¸Ð·ÑÑÐµÐ½Ð¸Ñ ÑÑÐ¾Ð³Ð¾ Ð±ÑÐ»Ð° ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð° Ð±Ð°Ð·Ð¾Ð²Ð°Ñ Ð¼Ð¾Ð´ÐµÐ»Ñ (ÑÐ²ÐµÑÑÐ¾ÑÐ½Ð°Ñ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½Ð°Ñ ÑÐµÑÑ) Ñ ÑÐ¾ÑÐ½Ð¾ÑÑÑÑ Ð±Ð¾Ð»ÐµÐµ 90% Ð½Ð° Ð½Ð°Ð±Ð¾ÑÐµ Ð´Ð°Ð½Ð½ÑÑ CIFAR-10 Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ ÑÐ»ÐµÐ´ÑÑÑÐ¸Ñ Ð¼ÐµÑÐ¾Ð´Ð¾Ð²: ÑÐºÐ°Ð»Ð° ÑÐºÐ¾ÑÐ¾ÑÑÐ¸ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ, ÑÐ¼ÐµÐ½ÑÑÐµÐ½Ð¸Ðµ Ð²ÐµÑÐ°, Ð³ÑÐ°Ð´Ð¸ÐµÐ½ÑÐ½Ð¾Ðµ Ð²ÑÑÐµÐ·Ð°Ð½Ð¸Ðµ, Ð½Ð¾ÑÐ¼Ð°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ Ð´Ð°Ð½Ð½ÑÑ, Ð½Ð¾ÑÐ¼Ð°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ Ð¿Ð°ÑÑÐ¸Ð¸, ÑÐ²ÐµÐ»Ð¸ÑÐµÐ½Ð¸Ðµ Ð´Ð°Ð½Ð½ÑÑ, ÐÑÑÐ°ÑÐ¾ÑÐ½ÑÐµ ÑÐ²ÑÐ·Ð¸. Ð¡ Ð¿Ð¾Ð¼Ð¾ÑÑÑ ÑÑÐ¾Ð¹ Ð¼Ð¾Ð´ÐµÐ»Ð¸ Ð±ÑÐ»Ð¾ Ð¿ÑÐ¾Ð²ÐµÐ´ÐµÐ½Ð¾ Ð½Ð°ÑÑÐ½Ð¾Ðµ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ðµ, Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð½Ð¾Ðµ Ð¸Ð·ÑÑÐµÐ½Ð¸Ñ ÑÑÑÐµÐºÑÐ¾Ð² Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð² ÑÐ²ÐµÑÑÐ¾ÑÐ½ÑÑ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÑÑ. ÐÑ ÑÐºÑÐ¿ÐµÑÐ¸Ð¼ÐµÐ½ÑÐ¸ÑÐ¾Ð²Ð°Ð»Ð¸ Ñ ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÐ¼Ð¸ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ°Ð¼Ð¸ Ð±Ð°Ð·Ð¾Ð²Ð¾Ð¹ Ð¼Ð¾Ð´ÐµÐ»Ð¸, Ð° Ð·Ð°ÑÐµÐ¼ ÑÐ°ÑÑÐ¼Ð¾ÑÑÐµÐ»Ð¸ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÑ Ð¼Ð¾Ð´ÐµÐ»Ð¸ Ð² Ð¾ÑÐ½Ð¾ÑÐµÐ½Ð¸Ð¸ Ð¿ÐµÑÐµÐ¾Ð±ÑÑÐµÐ½Ð¸Ñ, Ð½ÐµÐ´Ð¾Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ð¸ ÑÐ¾ÑÐ½Ð¾ÑÑÐ¸, Ð° ÑÐ°ÐºÐ¶Ðµ ÑÑÐ¾Ð±Ñ Ð¿Ð¾Ð½ÑÑÑ ÑÐ»Ð¾Ð¶Ð½ÑÑ Ð²Ð·Ð°Ð¸Ð¼Ð¾ÑÐ²ÑÐ·Ñ Ð¼ÐµÐ¶Ð´Ñ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ°Ð¼Ð¸ Ð¸ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÑÑ Ð¼Ð¾Ð´ÐµÐ»Ð¸. Ð ÐµÐ·ÑÐ»ÑÑÐ°ÑÑ Ð¿Ð¾ÐºÐ°Ð·ÑÐ²Ð°ÑÑ Ð¿Ð¾Ð²ÐµÐ´ÐµÐ½Ð¸Ðµ Ð±Ð°Ð·Ð¾Ð²Ð¾Ð¹ Ð¼Ð¾Ð´ÐµÐ»Ð¸ Ð¿ÑÐ¸ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ð¸ ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÑ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð², Ð° ÑÐ°ÐºÐ¶Ðµ Ð¿Ð¾ÐºÐ°Ð·ÑÐ²Ð°ÑÑ, ÐºÐ°Ðº ÑÐ»Ð¾Ð¶Ð½ÑÐµ Ð·Ð°Ð²Ð¸ÑÐ¸Ð¼Ð¾ÑÑÐ¸ Ð¼ÐµÐ¶Ð´Ñ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ°Ð¼Ð¸ Ð²Ð»Ð¸ÑÑÑ Ð½Ð° ÐºÐ¾Ð½ÐµÑÐ½ÑÑ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÑ Ð¼Ð¾Ð´ÐµÐ»Ð¸, Ð´ÐµÐ¼Ð¾Ð½ÑÑÑÐ¸ÑÑÑ Ð²Ð°Ð¶Ð½Ð¾ÑÑÑ Ð¿Ð¾Ð½Ð¸Ð¼Ð°Ð½Ð¸Ñ Ð²Ð»Ð¸ÑÐ½Ð¸Ñ ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð°ÑÐ¸Ð¹ Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð². ÐÐ° Ð¾ÑÐ½Ð¾Ð²Ð°Ð½Ð¸Ð¸ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ Ð¼Ð¾Ð¶Ð½Ð¾ ÑÐ´ÐµÐ»Ð°ÑÑ Ð²ÑÐ²Ð¾Ð´, ÑÑÐ¾ Ð´Ð»Ñ Ð´Ð¾ÑÑÐ¸Ð¶ÐµÐ½Ð¸Ñ Ð½Ð°Ð¸Ð²ÑÑÑÐµÐ¹ ÑÐ¾ÑÐ½Ð¾ÑÑÐ¸ ÑÑÐµÐ±ÑÐµÑÑÑ Ð¾Ð¿ÑÐ¸Ð¼Ð°Ð»ÑÐ½Ð¾Ðµ ÑÐ¾ÑÐµÑÐ°Ð½Ð¸Ðµ Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¾ÑÐ°, ÑÑÐ½ÐºÑÐ¸Ð¸ Ð°ÐºÑÐ¸Ð²Ð°ÑÐ¸Ð¸, ÑÐ°Ð·Ð¼ÐµÑÐ° Ð¿Ð°ÑÑÐ¸Ð¸ Ð¸ Ð¸Ð½Ð¸ÑÐ¸Ð°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸.
description: The theme of the final graduation research: Â«Evaluating the effects of hyper-parameters in convolutional neural networksÂ». This research is devoted to research the effects of neural networks hyper-parameters in Convolutional Neural networks using the CIFAR-10 dataset. The Tasks that were solved during the research: Review of hyperparameters and existing methods for understanding the effects of hyperparameters in convolutional neural networks. Development of a base model (convolutional neural network) using CIFAR-10 dataset. Interaction of hyperparameters in the developed convolutional neural network. Visualization of model performance using different hyperparameters. Analysis of the interaction of hyperparameters in convolutional neural networks. Understanding the impact of Convolutional Neural Networks hyperparameter combinations. The work was carried out based on an open dataset CIFAR-10. In the Python 3 programming language, with the pytorch and matplotlib libraries. Neural networks have many hyper-parameters, including choice of gradient descent method, activation functions, regularization, normalization, batch size, and so on. While best practices exist, interactions between different hyperparameter options can be difficult to predict. To study this, a base model (convolutional neural network) with more than 90% accuracy was developed on the CIFAR-10 dataset using the following techniques: Learning Rate Scale, Weight Decay, Gradient Cutting, Data Normalization, Batch Normalization, Data Augmentation, Residual Connections. Through this model, a scientific research was carried out dedicated to studying the effects of hyper-parameters in convolutional neural networks. We experimented with different hyper-parameters on the base model and then looked at the performance of the model with regard to overfitting, underfitting and accuracy and also to be able to understand the complicated relationship between hyper-parameters and model performance. The results show the behavior of the model when using different hyperparameters and also reveal how complicated dependencies between hyperparameters influence the final performance of the model, demonstrating that it is important to understand the impact of hyperparameter combinations. Based on the research, it can be concluded that an optimal combination of optimizer, activation function,batch size and initialization is required to reach the highest accuracy.