4-Bit Deep Learning
Tags: Computing, Neural Networks, Artificial Intelligence, Deep Learning, Data

4-Bit Computing

4-Bit Deep Learning

Powerful neural networks can be trained on smaller devices with faster speeds and less energy.

Bit needs Energy

Deep Learning is a very ineffective energy-consuming process because it requires a large amount of data and ample computational resources which is the reason for more energy consumption. This problem has become worse with the recent development to obtain an accurate model. The models have become larger and been trained on huge data points for days and will not stop since the data might appear in a new format which needs to be again trained.

Deep learning has a large carbon footprint.

There has been researches to who are trying to find algorithms that can use less data for training and hardware that can run the algorithm faster. IBM researchers are proposing on reducing the number of bits needed to represent the data to 4-bit size, which was presented in the NeurIPS the largest annual AI research conference, could increase the speed and cut the energy costs needed to train the deep learning model.

This also helps in training the powerful AI models possible on the smartphones and smaller computing devices, which helps in keeping the data local on the device and improving privacy for the end customer.

The computers stores and process data in the form of 1s and os and this fundamental unit of data is known as bits. This bits are combined into multiple strings of bits to represent the complete information. Two bits can represent four ways of representing the information because there are 2² combinations: 00, 01, 10, and 11. Four bits can represent 2⁴, or 16 ways of information. Eight bits can represent 2⁸, or 256. And so on.

In a 4-bit deep learning the 4-bit computer is able to represent the data using 4-bits. The data in the training process should be one of the 16 numbers between -8 and 7, since this are the only numbers the computer can represent. This is the same size for all the data provided to the neural network, used to represent the neural network, and the intermediate data to be stored during the training of the model.

To understand the above concept let us assume the training data is a black-and-white image. The image is converted to numbers for the computer to understand. Each pixel information is represented interms of the grayscale value 0 for black and 1 for white, the decimals can be used represent the shades of gray. the image is now a list of numbers between 0 and 1, which is represented between -8 and 7 in a 4-bit representation. The idea is to linearly scale the list of numbers, so value 0 becomes -8 and value 1 becomes 7, and the decimals map to the integers in the middle in the 4-bit representation.

Scale the number from 0 to 1 between -8 and 7, and then round any decimals to whole numbers

This may not be a accurate method since the value of 0.1 is -6.5 and would end up in representing a whole number by rounding -6.5 to -7. In this way you might be losing some precision in representing the image

Lower the number of bits the lesser the details of the photo

The neural network shown below contains nodes and connections. In a computer this turns into a series of numbers. Each of the node contains the activation value, which can range from 0 to 1 and the connections are called weight which can range from -1 to 1.

Neural Network

The weights and activations are scaled in the same way has pixels, but the activations and weights also change every round of the training. The activation ranges can be 0.2 to 0.9 in one round and 0.1 to 0.7 in the next round. In order to accommodate this changes the IBM group proposed to rescale those ranges to stretch between -8 and 7 in every round (as shown below), which effectively avoids losing too much precision.

Rescaling the activations and weights in the Neural Network

In order to represent the intermediate data that can grow during the training, the values can grow across several orders of magnitude, which is not like the values used to represent the images. The data can be either 0.001, or large like 1,000. The linear scaling between -8 and 7 cannot be used since it loses the granularity at the tiny end of the scale.

Linearly scaling numbers that span several orders of magnitude would lack of precision

The IBM researchers achieved by logarithmically scaling the intermediate numbers. The logarithmic scale you might recognize, with a so-called “base” of 10, using only four bits of complexity. (The researchers instead use a base of 4, because trial and error showed that this worked best.) From the scale we can see how it lets encode both tiny and large numbers within the bit constraints.

Logarithmic scale with base 10

By using all this information the latest paper published by IBM researchers by running several experiments where they simulate 4-bit training for a variety of deep-learning models in computer vision, speech, and natural-language processing. The final outcomes shows a limited loss of accuracy in the models’ overall performance compared with 16-bit deep learning. The process is also more than seven times faster and seven times more energy efficient.

Conclusion

Still there is a long way to go before 4-bit Deep Learing are put in practice. To implement in the real world there needs to be new 4-bit hardware designed and developed. The existing computing devices can be used to run faster and reduce the battery consumption, so that it can be deployed on smaller computing devices if it works. Many companies are switching the process of training the AI models, like speech-to-text and autocorrect systems, away from the cloud and onto user phones. This helps in the users’ privacy by keeping their data on their own phone while still using the device’s AI capabilities.

References