Generating-positive-feedbacks-for-restaurants-small-transform-model-vs-fine-tunning-Qwen3-
Collected almost 16,000 positive feedbacks from restaurants with score more than 4.7(out of 5) located in Moscow from https://yandex.ru/maps
🍽️Restaurant Feedback Generator
link to Github : https://github.com/mahdishahmari/Generating-positive-feedbacks-for-restaurants-small-transform-model-vs-fine-tunning-Qwen3-/tree/main
the aim was to check how far can local small transofmer model be pushed with limited parameters(6.350264 M parameters in this case).
model created based on MinGPT from https://github.com/karpathy/minGPT (karpathy Andrej) with small changes
then used same dataset and tried to fine tune Qwen3-0.6B model from https://huggingface.co/Qwen/Qwen3-0.6B for 2 Epochs
🎯 Key Features
Automatically generate positives reviews for mini-gpt added BPETokenizerSimple which creates vocab from data and tokenize it (the best was 3000 vocabs) also added RotaryPositionalEmbedding as its the effective way instead of The "original attention positional embedding" ==> Absolute Positions
Link to get Adapter saved parameters to load locally https://drive.google.com/file/d/1t6SDglyGEf09YvjMp_g9wHHkS-d3J3xq/view?usp=drive_link
Additional points:
1- Use file russian_comments_corrected-2.txt for miniGPT
2- Use file russian_comments_corrected.txt for Qwen3-Tunned or just get Qwen3.csv file and skip the step of proccessing data
3- For miniGPT with 4080 super it took around 80-90 minutes to train 120,000 epochs (did to be sure that cant get better result))! )
4- Fine tunning Qwen for data(which has 38296 lines) took 1 hour 25 minutes 2 epochs (fine tunned with Unsloth)
as shown below the fine tunned model gives much better output (as was expected) but it was cool to try both for me!
Qwen3-Tunned output :
Очень вкусная кухня, приятная атмосфера и персонал. Всем советую!
P.S : Delicious food, pleasant atmosphere and staff. I recommend it to everyone!
mini-gpt:
Повело на вкусный завтрак у дома: теперь всегда стоит подно общаться с первого раза рождения) Неплохая кухня, большой выбор блюд. Персонал - всегда прислушиваются и попахти бокалы раки хватает. Сегодня они покорили, видимо уютный хамит отлично пива и отменные в обламаете в осно м дома:!!!
P.S : use google translate to see the meaning but its not so good as it should be