title: ÐÐ½Ð°Ð»Ð¸Ð· ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÐ¸ Ð¿ÑÐ¸Ð¼ÐµÐ½ÐµÐ½Ð¸Ñ 16-ÑÐ°Ð·ÑÑÐ´Ð½ÑÑ ÑÐ¾ÑÐ¼Ð°ÑÐ¾Ð² Ð¼Ð°ÑÐ¸Ð½Ð½ÑÑ ÑÐ¸ÑÐµÐ» Ñ Ð¿Ð»Ð°Ð²Ð°ÑÑÐµÐ¹ ÑÐ¾ÑÐºÐ¾Ð¹ Ð¿ÑÐ¸ ÑÐµÑÐµÐ½Ð¸Ð¸ Ð·Ð°Ð´Ð°Ñ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ð¸ÑÐºÑÑÑÑÐ²ÐµÐ½Ð½ÑÑ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÐµÐ¹
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð±Ð°ÐºÐ°Ð»Ð°Ð²ÑÐ°
titleType: Subtitle
title: Analysis of the effectiveness of using 16-bit floating-point machine number formats when solving problems of training artificial neural networks
Creators: name: ÐÐ±Ð¾ÐµÐ½ÐºÐ¾Ð²Ð°, ÐÐ»Ð°Ð´Ð¸Ð»ÐµÐ½Ð°
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2024
Subjects: ieee 754
posit
bfloat16
Ð°ÑÐ¸ÑÐ¼ÐµÑÐ¸ÐºÐ¸ Ð¿Ð¾Ð½Ð¸Ð¶ÐµÐ½Ð½Ð¾Ð¹ ÑÐ¾ÑÐ½Ð¾ÑÑÐ¸
Ð¾Ð±ÑÑÐµÐ½Ð¸Ðµ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÐµÐ¹
low-precision arithmetics
training of neural networks

description: Ð Ð´Ð°Ð½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÐµ ÑÐ°ÑÑÐ¼Ð¾ÑÑÐµÐ½Ñ ÑÐ¾ÑÐ¼Ð°ÑÑ Ð¼Ð°ÑÐ¸Ð½Ð½ÑÑ ÑÐ¸ÑÐµÐ» Ñ Ð¿Ð»Ð°Ð²Ð°ÑÑÐµÐ¹ ÑÐ¾ÑÐºÐ¾Ð¹ Ð¿Ð¾Ð½Ð¸Ð¶ÐµÐ½Ð½Ð¾Ð¹ ÑÐ°Ð·ÑÑÐ´Ð½Ð¾ÑÑÐ¸ â FP16, BF16, Posit16 Ð¸ Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½ Ð°Ð½Ð°Ð»Ð¸Ð· ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÐ¸ Ð¸Ñ Ð¿ÑÐ¸Ð¼ÐµÐ½ÐµÐ½Ð¸Ñ Ð´Ð»Ñ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÐµÐ¹. ÐÑÐ»Ð¸ ÑÐµÑÐµÐ½Ñ ÑÐ»ÐµÐ´ÑÑÑÐ¸Ðµ Ð·Ð°Ð´Ð°ÑÐ¸: 1. ÐÐ·ÑÑÐµÐ½Ð¸Ðµ ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ñ ÑÐ°ÑÐ°ÐºÑÐµÑÐ¸ÑÑÐ¸Ðº ÑÐ¾ÑÐ¼Ð°ÑÐ¾Ð² Ð¼Ð°ÑÐ¸Ð½Ð½ÑÑ ÑÐ¸ÑÐµÐ» Ñ Ð¿Ð»Ð°Ð²Ð°ÑÑÐµÐ¹ ÑÐ¾ÑÐºÐ¾Ð¹: FP32, FP16, BF16 Ð¸ Posit16. 2. ÐÑÐ±Ð¾Ñ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð½ÑÑ ÑÑÐµÐ´ÑÑÐ² Ð¸ ÑÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ ÐºÐ¾Ð¼Ð¿Ð»ÐµÐºÑÐ° Ð¼Ð¾Ð´ÐµÐ»ÐµÐ¹ Ð²ÑÑÐ¸ÑÐ»ÐµÐ½Ð¸Ð¹ Ñ Ð¸ÑÑÐ»ÐµÐ´ÑÐµÐ¼ÑÐ¼Ð¸ ÑÐ¾ÑÐ¼Ð°ÑÐ°Ð¼Ð¸ Ð¼Ð°ÑÐ¸Ð½Ð½ÑÑ ÑÐ¸ÑÐµÐ». 3. ÐÑÐ¾Ð²ÐµÐ´ÐµÐ½Ð¸Ðµ ÑÐºÑÐ¿ÐµÑÐ¸Ð¼ÐµÐ½ÑÐ¾Ð² Ð½Ð° ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÑ ÑÐ¾ÑÐ¼Ð°ÑÐ°Ñ Ñ Ð¿Ð»Ð°Ð²Ð°ÑÑÐµÐ¹ ÑÐ¾ÑÐºÐ¾Ð¹ Ð´Ð»Ñ ÑÐµÑÐµÐ½Ð¸Ñ Ð·Ð°Ð´Ð°ÑÐ¸ ÐºÐ»Ð°ÑÑÐ¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¸ Ð¸Ð·Ð¾Ð±ÑÐ°Ð¶ÐµÐ½Ð¸Ð¹ Ð¸ Ð¿Ð¾ÑÐ»ÐµÐ´ÑÑÑÐµÐµ ÑÐµÑÑÐ¸ÑÐ¾Ð²Ð°Ð½Ð¸Ðµ Ð¿Ð¾Ð»ÑÑÐµÐ½Ð½ÑÑ ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐ¾Ð² Ð½Ð° Ð½Ð¾Ð²Ð¾Ð¼ Ð½Ð°Ð±Ð¾ÑÐµ Ð´Ð°Ð½Ð½ÑÑ. 4. ÐÑÐµÐ½ÐºÐ° Ð¸ ÑÑÐ°Ð²Ð½ÐµÐ½Ð¸Ðµ Ð¿Ð¾Ð»ÑÑÐµÐ½Ð½ÑÑ Ð¿Ð¾ÐºÐ°Ð·Ð°ÑÐµÐ»ÐµÐ¹ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÐ¸, ÑÐ°ÐºÐ¸Ñ ÐºÐ°Ðº ÑÐ¾ÑÐ½Ð¾ÑÑÑ Ð¿ÑÐµÐ´ÑÐºÐ°Ð·Ð°Ð½Ð¸Ð¹, Ð²ÑÐµÐ¼Ñ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ð¸ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ðµ Ð¿Ð°Ð¼ÑÑÐ¸, Ð²Ð¸Ð·ÑÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐ¾Ð². Ð Ð°Ð±Ð¾ÑÐ° Ð²ÑÐ¿Ð¾Ð»Ð½ÑÐ»Ð°ÑÑ Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ° PyTorch Ð¸ Ð±Ð¸Ð±Ð»Ð¸Ð¾ÑÐµÐº QPyTorch Ð¸ Qtorch+ Ð´Ð»Ñ ÑÐ°Ð±Ð¾ÑÑ Ñ Ð°ÑÐ¸ÑÐ¼ÐµÑÐ¸ÐºÐ°Ð¼Ð¸ Ð¿Ð¾Ð½Ð¸Ð¶ÐµÐ½Ð½Ð¾Ð¹ ÑÐ¾ÑÐ½Ð¾ÑÑÐ¸ Ð¸ ÑÐ¾ÑÐ¼Ð°ÑÐ¾Ð¼ Posit. Ð ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐµ Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ñ ÑÐ°Ð±Ð¾ÑÑ Ð±ÑÐ» Ð¿ÑÐ¾Ð²ÐµÐ´ÐµÐ½ ÑÑÐ´ ÑÐºÑÐ¿ÐµÑÐ¸Ð¼ÐµÐ½ÑÐ¾Ð² Ð¿Ð¾ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ð¸ÑÐºÑÑÑÑÐ²ÐµÐ½Ð½Ð¾Ð¹ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½Ð¾Ð¹ ÑÐµÑÐ¸ Ñ ÑÐ¾ÑÐ¼Ð°ÑÐ°Ð¼Ð¸ Ð¼Ð°ÑÐ¸Ð½Ð½ÑÑ ÑÐ¸ÑÐµÐ» Ñ Ð¿Ð»Ð°Ð²Ð°ÑÑÐµÐ¹ ÑÐ¾ÑÐºÐ¾Ð¹. ÐÐ¾Ð»ÑÑÐµÐ½Ñ Ð¾ÑÐµÐ½ÐºÐ¸ ÑÐ¾ÑÐ½Ð¾ÑÑÐ¸ (accuracy), Ð²ÑÐµÐ¼ÐµÐ½Ð¸ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ð¸ Ð·Ð°ÑÑÐ°Ñ Ð¿Ð°Ð¼ÑÑÐ¸.
description: This paper examines the low-precision floating-point machine number formats â FP16, BF16, Posit16 and analyzes the effectiveness of their use for training neural networks. The following tasks were solved: 1. Study the technical characteristics of floating-point formats: FP32, FP16, BF16 and Posit16. 2. Selection of software tools and implementation of a set of calculation models with the studied machine number formats. 3. Conducting experiments on various floating-point formats to solve the problem of image classification and then testing the results obtained on a new data set. 4. Evaluate and compare the obtained performance indicators, such as accuracy, training time and memory use, and visualize the results. The work was done using the PyTorch framework and the QPyTorch and Qtorch+ libraries for working with reduced precision arithmetic and the Posit format. As a result of the work, a number of experiments were carried out on training an artificial neural network with floating-point machine number formats. Estimates of accuracy, training time and memory costs were obtained.