Please, Translate the following English into Korean: 
So SmoothQuant makes the weight and activation into 8-bit, right, by smoothing the activations, pushing the quantization difficulty from the activation to the weight, improving the throughput for large language model. What about the batch size is small, okay, in the edge scenario or real-time interaction, like you want to talk to a robot, one-to-one conversation. When the batch size is small, we observe this is the real-time model, the x-axis is the compute intensity, the y-axis is the measured Tops per second. When the batch size is one, there is actually a pretty low utilization, this is your peak peak Tops per second, this is the measured Tops per second. So no matter what is underutilized, it's highly because it's highly memory bounded, right, because large language models are pretty big, like LAMMA 2.7b has 7 billion parameters. If you're using FP16, how many storage do you need? That's 14 gigabytes, right, 14 gigabytes, that's pretty big. You have to fetch 14 gigabytes in an autoregressive manner, which means generating each token requires 14 gigabytes of memory access. Remember, computation is cheap, memory is expensive, we want to reduce those memory footprint. Compared with the weight, the activation is actually small, right, so the weight is like 4k by 4k, but the activation, if it's single batch, is like 1 by 4k, it's a thousand times smaller. Therefore, we should focus on compressing the weight, so that's why we introduce this low bit weight only quantization. We're also going to see that in the homework. So that's why we need such weight only quantization to reduce the memory bandwidth. Previously, we have to fetch 14 gigabytes of memory, what about if we quantize the weight to 4 bit for LAMMA 7b? What is the memory footprint now? 3.5 gigabytes, 4 times smaller. So this is naively doing that from this is a weight matrix, so using the FP16 representation, the 8 by 4 matrix, this is the quantizer version, it's again 8 by 4, but it's quantized using 3 bit in this case, easy to show it, but immediately we see it's perplexity degradation. The lower the perplexity, the better the quality, but here unfortunately there's a huge jump of the perplexity if we naively quantize it. Perplexity is a measurement of the quality of a language model, it's showing the accuracy of predicting the next word, whether you're accurately predicting the next word or not. So the lower, the better. So unfortunately, directly doing the round to nearest quantization using 3 bit hurt the accuracy a lot. Even if we're using this group-wise quantization, so here every 128 numbers are quantized together and we have a shared scaling factor, a finer granularity of shared scaling factor every 128 elements, which we covered in the second lecture of this quantization part. Interestingly, we find not all the weights are equally important, just by quantizing 1%, keep 1% of the rows in FP16, it helps a lot immediately bring back the perplexity for the original value. This is so amazing, we seem to find a way to quantize that, right? Only keep 1% into FP16, like here, only one channel keep it the same as before, just don't quantize it, and immediately bring back the perplexity, the quality. So therefore we have two natural to-do's. One to-do is to think about how do we choose those saving channels, which channel is important. There exists only 1% of the channel, but how do we systematically select those 1% of the channels. The second to-do is keeping FP16 will make the inference kernel difficult. How do we get rid of this mix of precision and still use full quantization? Everything will be in blue rather than having 1% in yellow. So let's answer those two questions. When we are doing pruning, how do we select these important weights? Which one to prune, which one to remove? We look at the weight itself, right? If it is large, we think, oh, that's an important weight, important channel, we should keep it. What if we do it here? We have to wait. If it is large, we just keep it, otherwise we remove it. Unfortunately, the complexity is pretty high. We did it another way. Don't look at the weight. Since weight is multiplied with the activation, let's look at those activations. Since during smooth quant we find some activations are pretty huge, we want to preserve those outlier channels. So say this is outlier channel, this is a pretty big activation channel, and that is consistent for different inputs, different tokens. They are all big channels. And then if this channel is big in the activation, this corresponding weight is considered salient or important. We should keep them. So using this way, activation, not the weight, that's why we call it activation-aware, weight-only quantization. You're quantizing the weight, but you look at activation to determine which weight is salient. By looking at the activations magnitude, it's easier to recover the perplexity. So we solve the first to-do, and look at the activation, not the weight to determine those 1% salient channel. Then we have the second to-do. Can we not rely on this mixed precision, still use all-in-four or all-in-three to get rid of this mixed precision? We tried a very simple technique. We just multiply this weight channel by two, and divide the activation channel by two, so they are mathematically equal like smooth quant. So by multiplying it from like 1.5, 2, and 4, we find this is a very effective way to recover the perplexity. Without introducing this FP16, 1% of channels in FP16, just multiply that salient channel by a number larger than one, and bring back this perplexity. This is so amazing. Previously, we had to rely on 1% of the channels being unquantized in FP16, making the kernel difficult to write. But now, just multiply the salient channel by a larger number, and it will recover. But another problem came. How large should we multiply that channel? Right here, you first see the perplexity decrease and the increase. So there must be a sweet spot. We want to automatically. Search the multiplier. So let's see, analyze first why enlarging the channel make it easier to recover the perplexity. So this layer is denoted by weight times the activation, and then we care about the quantization error from the quantized version of the w times x. So Qw, quantized version of w, basically equal to, this is defining the range, and we divided the range by the number of centroids. You have n bits, they are to the power of n minus one centroids. And this is the distance between each centroid. And then we give w divided by this number and multiply this number to the outside, and we have to round it to the nearest integer. So that's the quantized version of w. If we, what if we scale that? Like here, we scale it by like 1.5 by two for those salient weight. What happened to that? So we scale up the weight, and we have to scale down the activation. So previously Qw times x now is Qw times s times x divided by s, so they are still mathematically equal. S gets cancelled here. If we plug in ws into the representation for the Q, we can see sw is here, since w becomes sw. And x divided by s is here. What happened here is that the rounding function always has an expectation of 0.25. Since rounding is always, the rounding error ranges from zero to 0.5, right? Like 1.5 gets rounded to two, 1.75 gets rounded to two as well. The average is 0.25, it's a quarter, since it's between zero and 0.5. So this doesn't change. What about this delta? So this delta is only dependent on the maximum of the weight. So there are a group of weights in the vertical dimension, and the group size is 128. Just scale up one channel, it's very unlikely to change the maximum value, unless one is 125, 128, you hit that max value. Otherwise you're not going to change it. So this delta is not going to change. But only this s is something, s is larger than one, so this error is scaled down. So when the s is greater than one, the error is scaled down. That's why scaling up, scaling up the sitting channel can achieve the same effect of making that channel to be fp, 16. Question here? One question about the scaling up. Is the scaling up after the quantization or before the quantization? So it's easier to quantize. So we scaled it up, and then the equation is very similar to smooth quant, making it easy for industry to put into products. Same infrastructure, you can do either smooth quant or AWQ. So here we times, w times s, x divided by s. This can be fused to the previous operation, or fused into the linear norm. And then we take a data-driven approach to do a fast grade search, to search the best scaling factor which is greater than one. And later follow-up work, even propose a learning-based method to use gradient descent to learn the best scaling factor. So this is 3-bit, group size 128 on LAMA, and also LAMA2. AWQ shows consistent better performance compared with the nearest, or GPTQ or GPTQR, on 7B all the way to 30B models. It also work well for multi-model and large language model, which we're going to introduce in the next lecture on region transformers. So this is Flamingo for image captioning. This is comparison with different baselines, actually pretty significant improvement about the accuracy here. Given this image, the round to nearest baseline quantization model says a model airplane is flying in the sky. AWQ can say two toy airplanes sitting on a grass field. This one baseline model is saying a man is holding a baby elephant in his arm versus AWQ says a man, his daughter, pulls with an elephant. And the last one, a man and a dog walking past some bushes versus AWQ, two dogs are walking on the street. You can even use LAMA, quantized LAMA, to do visual reasoning. Given this, there is some caption here, but this is all represented in the image format, although it has some text, you have to automatically do OCR to understand it. And you can make a world map. The baseline quantization RTA model says there are small pictures of the Earth and other planets placed on top of the food versus AWQ says a lighthearted and humorous take on the concept of looking at the pieces of the Earth from space. Plate of fried food, especially chicken nuggets, is presented with the caption. And the caption is actually exactly the same as the caption here. So it means this vision language model is automatically doing the OCR, understand the text here. One more example, able to recognize who is painting this, Leonardo da Vinci. So, smooth quant and AWQ are widely used these days, that's why we also put it in the homework in the lab 4, which we released last week. We'll give you the code and also it will pave the way for lab 5, which we are going to actually implement that on our laptop. MediaFasterTransformer and TensorRTLM, this is the library run large language model inference, which is actually pretty amazing library released actually last week. How amazing, how timely we are. TensorRTLM, they are using the smooth quant and AWQ as their quantization approaches. Intel will train Berkeley's BLM, we are actually going to talk about that, Berkeley FastChat. It's a hugging face and since time, several open source communities have been using that. So how do we translate this theoretical savings into measured speedup? And can we deploy this large language model on edge devices, like on our laptops, our phones? So I'm going to introduce PennyChat, which is a lightweight chatbot for large language model on the edge. This is what we designed, 3D printed computer which has a BESEN ARRI nano inside. And we also have a demo here on the right. So deploying large language model on the edge is quite useful. For example, you run CoPilot locally on your edge device, code completion, office, game chat. Especially coding, enterprise data is privacy sensitive, you don't want to upload to the cloud. But these devices are very resource constrained. Here, it's only a small BESEN ARRI nano. Resource constrained, low power, and also do not have access to the internet always. Privacy is important. So here is our TinyChat computer, you can ask questions and also scroll up to see the previous answers. So basically, TinyChat implements this 4-bit compressed AWQ model. The weight is 4-bit to save the memory footprint. And here we are running it on different laptops. This is running the CodeLlama, writing code using CodeLlama, pretty fast. And this is comparing MIT and Harvard, used different prompts. Blazing fast. After lab 4, you are going to implement something similar on your laptop, lab 5. And feel free to continue improving that and pushing to the code base. TinyChat engine is the final project, which is open-ended, but this could be one of the choices. And a key technique to enable such fast inference is algorithm and system co-design. On the algorithm side, it 4-bit AWQ quantization. On the system side, it's a tiny engine and parallel computing techniques. We introduced loop priority, blocking, cache locality, and also multi-threading, CUDA programming, different techniques. And also how do we lay out the 4-bit weight in memory and run time decoded from 4-bit to 16-bit to avoid this decoding overhead. That is also one of the key techniques to make it run fast. This is comparing on the 4090 GPU without and with AWQ. How much speed up you can get. So the FP16 version is throwing both the weight and activation in FP16. So that's 50 tokens per second running on 4090 GPU. This is the AWQ version. The weight is only 4-bit, activation is still 16-bit. Since this is the memory bottleneck, not the activation. So we keep activation in FP16 to preserve the accuracy. And we run time decode the weight from int4, to FP16, and do the arithmetic in FP16. Since computing is cheap, memory footprint is expensive. This is exactly the way we did in the efficient inference engine ISCA16. Now this method reborn and proved to be quite helpful to accelerate this large language model for real-time inference. This one already finished. This one is still slowly making progress. TinyChat is also flexible, support a lot of different large language models. Like MPT7B, like Nozick, Vulkan 7B, and also Vekuna 7B. It can also run the 13B parameter model on a MacBook. Even on the JSON-ORIN, which is a mobile GPU, 30 tokens per second running on LAMA2.