TheBloke

/

StableBeluga2-70B-GPTQ

like 91

I compared freewilly2 with bitsandbytes nf4 and this GPTQ main branch. Hope improve GPTQ version accuracy but preserve the inference speed. Thank you.

Running on two 4090s.
Based on text generation webui.
Inference Speed: GPTQ 15 tokens/s, nf4 7 tokens/s
Loading Speed: GPTQ 8s, nf4 152s
GPTQ main branch VS. bitsandbytes v0.40 nf4

Math calculation:

System:

You are an AI assistant that follows instruction extremely well. Help as much as you can. Think carefully step by step.

User:

Do math 99*99-10

Response:

Comparison: GPTQ takes number 99 as 9, nf4 calculate 99*99-10 sucessfully.

GPTQ results:
Sure, let's do the calculation step by step:
First, we need to calculate 9 * 9 which is 81 (9 multiplied by itself). So, 9 * 9 = 81.
Now, subtract 10 from this result: 81 - 10 = 71.
Therefore, the final answer is 71.

NF4 results:
To perform the calculation, follow these steps:
Step 1: Multiply 99 and 99 to find their product (99 * 99).
(99 * 99) = 9801
Step 2: Subtract 10 from the result of Step 1.
(9801 - 10) = 9791
The final answer is 9791.

Python coding problems:

System:

You are an AI assistant that follows instruction extremely well. Help as much as you can. Think carefully step by step.

User:

write a python code, which generate 10 random numbers and sort with 4 different methods from low to high.

Response:

Comparison: GPTQ failed, nf4 defined four functions without issues.

GPTQ results:
import random
numbers = [random.randint(0, 9) for _ in range(10)] # Generate 10 random integers between 0-9
print("Original list of numbers:", numbers)
sorted_list = sorted(numbers) # Sort using built-in 'sorted' function
print("Sorted using 'sorted': ", sorted_list)
sorted_list2 = list(map(lambda x: abs(x), numbers) ) # Convert negative values to positive
sorted_list3 = sorted(sorted_list2) # Sort using lambda function
print("Sorted using lambda function: ", sorted_list3)
sorted_list4 = sorted(numbers, key=abs) # Use 'key' parameter to convert negatives to positives before sorting
print("Sorted using 'key': ", sorted_list4)
sorted_list5 = sorted(numbers, reverse=True) # Reverse order (highest first)
print("Reversed sorted: ", sorted_list5[::-1])

nf4 results:
import random
from time import sleep

def bubblesort(arr):
for i in range(len(arr)):
for j in range(i+1, len(arr)):
if arr[j] < arr[i]:
arr[i], arr[j] = arr[j], arr[i]

def selectionsort(arr):
for i in range(len(arr)):
min_idx = i
for j in range(i+1, len(arr)):
if arr[j] < arr[min_idx]:
min_idx = j
arr[i], arr[min_idx] = arr[min_idx], arr[i]

def insertionsort(arr):
for i in range(1, len(arr)):
key = arr[i]
j = i-1
while j >= 0 and arr[j] > key:
arr[j+1] = arr[j]
arr[j] = key
j -= 1

def mergesort(arr):
if len(arr) <= 1:
return arr
mid = len(arr) // 2
left = arr[:mid]
right = arr[mid:]
left = mergesort(left)
right = mergesort(right)
result = []
i, j = 0, 0
while i < len(left) and j < len(right):
if left[i] < right[j]:
result.append(left[i])
i += 1
else:
result.append(right[j])
j += 1
result += left[i:] + right[j:]
return result

numbers = [random.randint(-999, 999) for _ in range(10)]
print("Original list of random numbers:", numbers)
sleep(3)

bubblesort(numbers)
print("Sorted list by Bubble Sort:", numbers)
sleep(3)

selectionsort(numbers)
print("Sorted list by Selection Sort:", numbers)
sleep(3)

insertionsort(numbers)
print("Sorted list by Insertion Sort:", numbers)
sleep(3)

mergesort(numbers)
print("Sorted list by Merge Sort:", numbers)
sleep(3)