아래 내용이 설명하는 것은 무엇인가요?

large-range models are very powerful, however they are very expensive as well. So these models have been widely deployed in today's everyday applications such as chatbots and software development. However, you usually notice that there are some messages from the websites of OpenAI and Anthropic saying that currently we are having very high demand and we cannot provide the service as you expected. So definitely one of the largest challenges in terms of deploying the large-range models is that they have really large sizes. So despite being very powerful, large-range models have outgrown the pace of the hardware capacity growth, especially like what is demonstrated in this figure. So basically the largest model is the one that is the largest in the world. The largest GPU available now is the Nvidia B200 GPU. So it has only 192GB of memory. However, the largest transformer model that has been publicly announced by Jensen Huang in GTC 2024 actually has 1.8 trillion parameters. So it is necessary for us to use some techniques to reduce the model sizes and make sure they are deployable on real devices. So naturally quantization can help reduce the deployment costs. Because if you want to serve a LAMA3, a CES, or a T-Series model, you need to have a lot of data. So if you want to serve a LAMA3, a 70 billion model, it actually requires 140GB of memory if you just directly serve this model in FP16. However, if you are able to quantize this model to INT8 in terms of the weights or INT4, you are able to significantly reduce the deployment costs from like 2 A100 GPUs with 80GB of memory all the way down to 1 A100 GPU with 40GB of memory, which is actually also much cheaper. So previously, our group has done some pioneering work. In terms of the quantization of large range models, which is called SmoothCount. So the idea of SmoothCount is that the large range models, their quantization difficulty actually is very different for the weights and activations. So specifically, the activations are very hard to quantize due to the number of outliers presented. So we are very lucky we can actually transfer the quantization difficulty from activation to weights through an operation called smoothing, which is multiplying the weights by scaling factors. So we can actually scale the factor that is larger than 1 and then multiply the inverse of this scaling factor to the activations. So W8A8 quantization is definitely good. It reduces the cost for the memory required and improves the peak throughput of large range model serving, since the models can be mapped onto INT8 and so forth with like 2x higher throughput. However, is W8A8 quantization enough for large range model deployment, especially on the edge? The answer is no. So basically, in this figure, we actually presented the roofline model for the RTX 4090 GPU, which is very common in the desktop AI applications. So basically, the x-axis of this figure is the computation intensity of the workload. So here we use the general matrix multiplication as an example. So here the computation intensity in terms of max per element is equivalent to the batch size of the general workload. So in this example, we can clearly see that there is a demarcation point at batch size equal to roughly 80 for this specific device. So this means that if the batch size goes beyond 80, then W8A8 quantization is more beneficial because the peak throughput of the INT8 tensor core is much higher than that of the FP16 tensor core. However, if we go below this value, we are entering the memory bound region. So in this case, the peak throughput of the, you know, the bare tensor cores actually does not. So this means that the entire workload is guaranteed by the peak performance of the entire workload. So from a more global point of view, we actually break down the large-range model inference into two stages. The first stage is called the contact stage, which is actually much faster than the generation stage or the autoregressive decoding stage. That is because the contact stage can actually be run in one pass using very high power of the man can be mapped onto the tensor cores for efficient computation. However, when it comes to the generation stage, as I have just mentioned, the process is actually autoregressive and tokens are generated in a one by one manner. So this means that our computation intensity is just one max per element. So basically, this lies in the sweet region of the weight only quantization instead of the weight and activation quantization, because the memory bandwidth is the more important factor for the final performance of the entire workload. So in terms of the memory traffic, we actually also break down it into the weight traffic and activation traffic. And not very surprising because the large-range models are very large in terms of their weights, and the batch size is very small, so the weight traffic dominates the entire traffic. So put all these things together to accelerate large-range models on the edge for the batch size equal to one inference, it is very important for us to reduce the weight traffic, so it becomes very natural for us to design algorithms that performs weight only quantization.