asiansoul/SmartLlama-3-Ko-8B-256k-PoSE-GGUF

🇰🇷 SmartLlama-3-Ko-8B-256k-PoSE

SmartLlama-3-Ko-8B-256k-PoSE is an advanced AI model that integrates the capabilities of several advanced language models, designed to excel in a variety of tasks ranging from technical problem-solving to multilingual communication, especially with its extended context length of 256k tokens. This model is uniquely positioned to handle larger and more complex datasets and longer conversational contexts, making it ideal for deep learning applications requiring extensive text understanding and generation.

📕 Merge Details

Component Models and Contributions

• NousResearch/Meta-Llama-3-8B and Meta-Llama-3-8B-Instruct: These models provide a solid foundation for general language understanding and instruction-following capabilities.
• winglian/llama-3-8b-256k-PoSE: Utilizes Positional Skip-wise Training (PoSE) to extend Llama's context length to 256k, significantly improving the model's ability to handle extensive texts and complex instructions, enhancing performance in tasks requiring long-duration focus and memory.
• Locutusque/Llama-3-Orca-1.0-8B: Specializes in mathematical, coding, and writing tasks, bringing precision to technical and creative outputs.
• abacusai/Llama-3-Smaug-8B: Improves the model's performance in real-world, multi-turn conversations, which is crucial for applications in customer service and interactive learning environments.
• beomi/Llama-3-Open-Ko-8B-Instruct-preview: Focuses on improving understanding and generation of Korean, offering robust solutions for bilingual or multilingual applications targeting Korean-speaking audiences.

🖼️ Key Features

• Extended Context Length: Utilizes the PoSE (Positional Encoding) technique to handle up to 256,000 tokens, making it ideal for analyzing large volumes of text such as books, comprehensive reports, and lengthy communications.

• Multilingual Support: While primarily focused on Korean language processing, this model also provides robust support for multiple languages, enhancing its utility in global applications.

• Advanced Integration of Models: Combines strengths from various models including NousResearch's Meta-Llama-3-8B, the instruction-following capabilities of Llama-3-Open-Ko-8B-Instruct-preview, and specialized capabilities from models like Llama-3-Smaug-8B for nuanced dialogues and Orca-1.0-8B for technical precision.

🎨 Models Merged

The following models were included in the merge:

• winglian/llama-3-8b-256k-PoSE: Extends the context handling capability. This model uses Positional Skip-wise Training (PoSE) to enhance the handling of extended context lengths, up to 256k tokens.
• Locutusque/Llama-3-Orca-1.0-8B: Enhances abilities in handling technical content. Specialized in computational, scientific, and technical tasks, improving the model's ability to process complex academic and technical language.
• abacusai/Llama-3-Smaug-8B: Improves multi-turn conversational abilities. Boosts performance in engaging in lengthy, context-aware dialogues necessary for effective customer service and interactive learning.
• beomi/Llama-3-Open-Ko-8B-Instruct-preview: Provides enhanced capabilities for Korean language processing. This model is fine-tuned to understand and generate Korean, making it ideal for applications targeting Korean-speaking users.
• NousResearch/Meta-Llama-3-8B-Instruct: Offers advanced instruction-following capabilities. It is optimized to follow complex instructions, enhancing the model's utility in task-oriented dialogues and applications that require a high level of understanding and execution of user commands.

🖋️ Merge Method

• DARE TIES: This method was employed to ensure that each component model contributes effectively to the merged model, maintaining a high level of performance across diverse applications. NousResearch/Meta-Llama-3-8B served as the base model for this integration, providing a stable and powerful framework for the other models to build upon.

💻 Ollama

ollama create smartllama-3-Ko-8b-256k-pose -f ./Modelfile_Q5_K_M

[Modelfile_Q5_K_M]

FROM smartllama-3-ko-8b-256k-pose-Q5_K_M.gguf
TEMPLATE """
{{- if .System }}
system
<s>{{ .System }}</s>
{{- end }}
user
<s>Human:
{{ .Prompt }}</s>
assistant
<s>Assistant:
"""

SYSTEM """
친절한 챗봇으로서 상대방의 요청에 최대한 자세하고 친절하게 답하자. 길이에 상관없이 모든 대답은 한국어(Korean)으로 대답해줘.
"""

PARAMETER temperature 0.7
PARAMETER num_predict 3000
PARAMETER num_ctx 256000
PARAMETER stop "<s>"
PARAMETER stop "</s>"

💻 Ollama Python Summarizing Normal Test Code

install all of these libraries

pip install requests beautifulsoup4 PyPDF2 langchain-community langchain

pose_test.py

import sys
import os
import requests
from bs4 import BeautifulSoup
import PyPDF2
from langchain_community.chat_models import ChatOllama
from langchain.schema import AIMessage, HumanMessage, SystemMessage

def clean_output(text):
    text = text.replace("</s>", "").strip()
    return text

def invoke_model(text):
    messages = [
        SystemMessage(content='You are an expert copywriter with expertise in summarizing documents.'),
        HumanMessage(content=f'Please provide a short and concise summary of the following text:\nTEXT: {text}')
    ]
    
    try:
        llm = ChatOllama(model="pose:latest")
        summary_output = llm.invoke(messages)
        if isinstance(summary_output, AIMessage):
            cleaned_content = clean_output(summary_output.content)
            return cleaned_content
        else:
            return "Unexpected data type for model output."
    except Exception as e:
        print(f"An error occurred while processing the model output: {str(e)}")
        return None

def fetch_text_from_url(url):
    try:
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')
        content = soup.find('div', {'id': 'bodyContent'})
        paragraphs = content.find_all('p')
        text_content = ' '.join(p.text for p in paragraphs)
        return text_content
    except requests.RequestException as e:
        print(f"Failed to fetch data from URL: {str(e)}")
        return None

def read_text_file(file_path):
    with open(file_path, "r", encoding="utf-8") as file:
        return file.read()

def read_pdf(file_path):
    with open(file_path, "rb") as file:
        reader = PyPDF2.PdfReader(file)
        text_content = ""
        for page in reader.pages:
            extracted_text = page.extract_text()
            if extracted_text:
                text_content += extracted_text + "\n"
        return text_content

def summarize_content(source):
    if source.startswith(('http://', 'https://')):
        text_content = fetch_text_from_url(source)
    else:
        _, file_extension = os.path.splitext(source)
        if file_extension.lower() == '.pdf':
            text_content = read_pdf(source)
        elif file_extension.lower() in ['.txt', '.text']:
            text_content = read_text_file(source)
        else:
            print("Unsupported file type")
            return
    
    if text_content:
        summary = invoke_model(text_content)
        print("Summary of the document:")
        print(summary)
    else:
        print("No text found or unable to extract text from source.")

if __name__ == '__main__':
    if len(sys.argv) < 2:
        print("Usage: python script.py <file_path_or_url>")
    else:
        source = sys.argv[1]
        summarize_content(source)

run txt file (assume txt is a.txt)

python pose_test.py a.txt 

run url (assume txt is url)

python pose_test.py url

You can find both test results below on the section : Test Result1

💻 Ollama Python Summarizing Test Code for the target lang response

install all of these libraries

pip install requests beautifulsoup4 PyPDF2 googletrans==4.0.0-rc1 langchain-community langchain aiohttp asyncio aiofiles

pose_lang.py

import sys
import os
import aiohttp
import PyPDF2
from bs4 import BeautifulSoup
from langchain_community.chat_models import ChatOllama
from langchain.schema import AIMessage, HumanMessage, SystemMessage
from googletrans import Translator
import logging
import asyncio
import aiofiles

# Setup logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')

def clean_output(text):
    """Cleans the model output text."""
    text = text.replace("</s>", "").strip()  # Specific cleaning operation
    return text

def translate_text(text, src_lang, dest_lang):
    """Translates text from source language to destination language using Google Translate."""
    if src_lang == dest_lang:
        return text
    translator = Translator()
    try:
        translation = translator.translate(text, src=src_lang, dest=dest_lang)
        return translation.text
    except Exception as e:
        logging.error(f"Translation failed: {e}")
        return text

def detect_language(text):
    """Detects the language of the given text."""
    translator = Translator()
    try:
        detected = translator.detect(text)
        return detected.lang
    except Exception as e:
        logging.error(f"Language detection failed: {e}")
        return None

async def invoke_model(text, target_lang):
    """Asynchronously invokes the chat model and processes the response with language-specific instructions."""
    llm = ChatOllama(model="pose:latest")
    try:
        # Define messages based on target language
        if target_lang == 'ko':
            messages = [
                SystemMessage(content='문서의 핵심 요약을 상세하게 제공해 주실 전문가로서, 다음 문서를 요약해 주세요.'),
                HumanMessage(content=f'다음 텍스트에 대한 전문적 요약을 제공해 주세요. 요약은 한국어의 언어적 뉘앙스에 맞게 최고 수준의 명확성과 세부 사항을 준수해야 합니다:\n\nTEXT: {text}')
            ]
        else:  # default to English if not Korean
            messages = [
                SystemMessage(content='As an adept summarizer, your expertise is required to condense the following document into its essential points in detail.'),
                HumanMessage(content=f'Kindly provide an expert summary of the text below, adhering to the highest standards of clarity and detail. Ensure the response is tailored to the linguistic nuances of English:\n\nTEXT: {text}')
            ]

        # Since invoke is not awaitable, run it in a thread if it's blocking
        response = await asyncio.to_thread(llm.invoke, messages)
        if isinstance(response, AIMessage):
            cleaned_content = clean_output(response.content)
            content_lang = detect_language(cleaned_content)
            print(f"Current content language: {content_lang}, Target language to be translated to: {target_lang}")
            if content_lang != target_lang:
                return translate_text(cleaned_content, content_lang, target_lang)
            return cleaned_content
        else:
            raise ValueError("Model did not return an AIMessage")
    except Exception as e:
        logging.error(f"Error during model invocation: {e}")
        return "Model invocation failed."


async def fetch_text_from_url(url):
    """Asynchronously fetches and extracts text content from a given URL."""
    async with aiohttp.ClientSession() as session:
        try:
            async with session.get(url) as response:
                content = await response.text()
                soup = BeautifulSoup(content, 'html.parser')
                main_content = soup.select_one('#mw-content-text, #bodyContent, .content')
                if not main_content:
                    logging.error("No content found in the expected sections.")
                    return None
                text_content = ' '.join(p.get_text() for p in main_content.find_all(['p', 'li'], string=True))
                return text_content
        except Exception as e:
            logging.error(f"Error fetching URL content: {e}")
            return None

async def read_text_file(file_path):
    """Asynchronously reads text from a text file."""
    async with aiofiles.open(file_path, mode='r', encoding='utf-8') as file:
        text_content = await file.read()
    return text_content

async def read_pdf(file_path):
    """Asynchronously reads text from a PDF file."""
    def sync_read_pdf(path):
        try:
            with open(path, "rb") as file:
                reader = PyPDF2.PdfReader(file)
                return ' '.join(page.extract_text() for page in reader.pages if page.extract_text())
        except Exception as e:
            logging.error(f"Error reading PDF file: {e}")
            return None

    return await asyncio.to_thread(sync_read_pdf, file_path)

async def summarize_content(source, language):
    """Processes input source (URL, file, text) and outputs a summary in the specified language asynchronously."""
    print("Processing input...")
    text_content = None
    if source.startswith(('http://', 'https://')):
        print("Fetching content from URL...")
        text_content = await fetch_text_from_url(source)
    elif os.path.isfile(source):
        _, file_extension = os.path.splitext(source)
        if file_extension.lower() == '.pdf':
            print("Reading PDF...")
            text_content = await read_pdf(source)
        elif file_extension.lower() in ['.txt', '.text']:
            print("Reading text file...")
            text_content = await read_text_file(source)
        else:
            print("Unsupported file type")
            return
    else:
        print("Unsupported file type")
        return

    if text_content:
        print("Summarizing content...")
        summary = await invoke_model(text_content, language)
        print("\n--- Summary of the document ---\n")
        print(summary)
    else:
        print("No text found or unable to extract text from source.")

if __name__ == '__main__':
    if len(sys.argv) < 3:
        print("Usage: python script.py <file_path_or_url_or_text> <language>")
        print("Language should be 'ko' for Korean or 'en' for English.")
    else:
        source = sys.argv[1]
        language = sys.argv[2]
        asyncio.run(summarize_content(source, language))

run txt file (assume txt is a.txt)

Korean response : python pose_lang a.txt ko
English response : python pose_lang a.txt en

run pdf file (assume pdf is a.pdf)

Korean response : python pose_lang a.pdf ko
English response : python pose_lang a.pdf en

run url (assume url is wikepedia)

Korean response : python pose_lang url ko
English response : python pose_lang url en

I added additional Google Translator here. If you request an answer in Korean and the answer is in English sometimes for the lang hallucination, this function detects it and answers you in Korean. Conversely, if you request a response in English and the response is in Korean for the lang hallucination, this function detects it and responds in English.

You can find both test results below on the section : Test Result2 for target lang response

🗞️ Configuration

The YAML configuration for this model:

models:
  - model: NousResearch/Meta-Llama-3-8B
    # Base model providing a general foundation without specific parameters
  - model: NousResearch/Meta-Llama-3-8B-Instruct
    parameters:
      density: 0.60
      weight: 0.25
  - model: winglian/llama-3-8b-256k-PoSE
    parameters:
      density: 0.60
      weight: 0.20
  - model: Locutusque/Llama-3-Orca-1.0-8B
    parameters:
      density: 0.55
      weight: 0.15
  - model: abacusai/Llama-3-Smaug-8B
    parameters:
      density: 0.55
      weight: 0.15
  - model: beomi/Llama-3-Open-Ko-8B-Instruct-preview
    parameters:
      density: 0.55
      weight: 0.30

merge_method: dare_ties
base_model: NousResearch/Meta-Llama-3-8B
parameters:
  int8_mask: true
dtype: bfloat16

Test OS Condition

Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,2
      Chip: Apple M1 Max
      Total Number of Cores: 10 (8 performance and 2 efficiency)
      Memory: 64 GB
      System Firmware Version: 10151.101.3
      OS Loader Version: 10151.101.3

🎊 Test Result1 (Normal)

SmartLlama-3-Ko-8B-256k-PoSE Summary Ability

consideration

Long sentences seemed to summarize well, but I observed that answers came in English. And when I asked for it to be translated into Korean, I confirmed that it was translated well. The summary seems to work well, but you can take into account the fact that there are times when it cannot be summarized directly in Korean.

Summary of Britney Spears on Wikipedia

Summary of Steve Jobs Text File

Summary of Jay Park on Wikipedia

🎊 Test Result2 (Target Language Summary Return)

SmartLlama-3-Ko-8B-256k-PoSE Summary Ability

consideration

I added additional Google Translator here. If you request an answer in Korean and the answer is in English, this function detects it and answers you in Korean. Conversely, if you request a response in English and the response is in Korean, this function detects it and responds in English.

If you don't get a clear answer, try running it several times.

Summary of economy pdf

python final2.py economy.pdf ko

# if you want english summary, en

Summary of Steve Jobs Text File

python final2.py steve.txt ko

# if you want english summary, en

Summary of Jay Park on Wikipedia

python final2.py https://en.wikipedia.org/wiki/Jay_Park ko 

# if you want english summary, en

Test Source From

박재범 - wikipedia - EN

박재범 - wikipedia - KR

Britney Spears - wikipedia - EN

한국은행 경제전망 보고서 - KR

[Community member : Mr Han' steve jobs txt file]

⛑️ Test Issue

2024-05-02

If you use load_summarize_chain(), there will be repetition. -> community member Mr.Han issue

Is it a merge issue? He thinks the merge target may be the issue.

chain = load_summarize_chain(
     llm,
     chain_type='stuff',
     prompt=prompt,
     verbose=False
)
output_summary = chain.invoke(docs)

-> investigating for me how to solve.....

Mr.Han is investgating the symptoms
Your OS is using REDHAT. Even if I run the code using the LLAMA3 model provided by ollama, there is an error.

I wonder if I should wait a little longer for Red Hat...

<|eot_id|><|start_header_id|>assistant<|end_header_id|>, ... omitted
Ha ha, thanks for the chat! You too have a great day and happy summarizing if you need it again soon!<|eot_id|><|start_header_id|>assistant<|end_header_id|>

It's not a merge problem... I think it's a fundamental problem that doesn't fit the OS environment... so I'm sharing it with you. Is there anyone who has the same problem as me in redhat?