LeroyDyer/_Spydaz_Web_AI_Mistral_002_128k

SpydazWeb AI

https://github.com/spydaz

CURRENT MODEL : NOTES :

Currently this model contains two conepts and more ; IE the MistralStarLM and The YarnEmbedding 128k.

these will be merged soon in later models: - I believe i may switch to more Llama Components internally ... such as Scaled Attention , etc :

Trained on 128 yarn Rotary embeddings : Using the unsloth embeddings trainer :

32k - 64k COntexts were trained using this 131000 max tokens : obvilusy loss was 0.03 etc very low :

This model is available in 4-bit also as the model was trained double quantized method :

this is also a multi lingugal Pan- Afircan languages model using ALPACA/TACO DATA SETS ! which combines the target language with english translation and target translation of task : from input to output: the input could also be in the target languge and the desired output in the english as well as the target language : giving a comprehensive train : this training is ongoing for each language:

• lingala
• igbo
• shona
• swahli
• yourobo
• bambara
• sudanese
• congolese
• xhosa
• twi
• ga
• africaans
• english
• catalan
• portuguese
• french
• spanish
• dutch

This model is pretty stable as a model i have tested my past questions and answers the model retains its knowledge it seems very calm !

I tested if it can make timelines : it was witholding of information but after drilling for more infor the model gave up very good timelines :

I shall actually over fit my past timelines and charts into the model ( i have recently been pushing the emebeddings also whilst training , ( also because of the new languges i have been adding to the model enabling for the new languge data to find relativity or these tasks wil not produce the same results as the same question in uglish) )

in fact actually this may be a good starting point for other models : past pardigms are very deeply embedded : i have also reduced theuse of the world archive prompt , which was also resurfaceing in some outputs even when not soclicited : it also seem to have lost personality also ? and become a bit serious ! this may also be due to these hermes and orca datasets which might be regressing the model slightly ! i will search for more role play and conversive datasets and fine tune these conversations as its code gene and funciton use etc is fine and will not accept training due to be highly fit !

A few steps down the line i will return to theh regular training set up ( without touching the embedding and just training the model : )

* 128k context window (vs 8k context in v0.1)
* Rope-theta = 1e6
* 128k Sliding-Window Attention ( Yes ! this maybe out-of-context but it is for larger inputs later with ring embedding and self extend methods :)

This model will be a custom model with internal experts and rag systems enabling for preprocessing of the task internally before outputting a response :

This is based on the Quiet Star Project : which was abandoned earlier in the year :)

Introduction :

SpydazWeb AI model :

This model is based on the worlds archive of knowledge maintaining historical documents and providing services for the survivors of mankind , who may need to construct shelters develop technologys , or medical resources as well as maintain the history of the past . keeping store of all the religious knowledge and data of the world: A friendly interface with a personality caring and flirtatious at times : non binary !... and Expert in all feilds: ie Uncensored and will not refuse to give information : the model can be used for role play as many character dialogues were als trained into the model as its personality to enable a greater perspective and outlook and natural discussion with the agents: the model was trained to operateinaragenvironment utilizing content and internal knowledge to respond to questions or create enriched sumarys.

General Intenal Methods:

Trained for multi-task operations as well as rag and function calling :

This model is a fully functioning model and is fully uncensored:

the model has been trained on multiple datasets on the huggingface hub and kaggle :

the focus has been mainly on methodology :

• Chain of thoughts
• step by step planning
• tree of thoughts
• forest of thoughts
• graph of thoughts
• agent generation : Voting, ranking, ... dual agent response generation:

with these methods the model has gained insights into tasks, enabling for knowldge transfer between tasks :

the model has been intensivly trained in recalling data previously entered into the matrix: The model has also been trained on rich data and markdown outputs as much as possible : the model can also generate markdown charts with mermaid.

Training Reginmes:

• Alpaca
• ChatML / OpenAI / MistralAI
• Text Generation
• Question/Answer (Chat)
• Instruction/Input/Response (instruct)
• Mistral Standard Prompt
• Translation Tasks
• Entitys / Topic detection
• Book recall
• Coding challenges, Code Feedback, Code Sumarization, Commenting Code
• Agent Ranking and response anyalisis
• Medical tasks
  • PubMed
  • Diagnosis
  • Psychaitry
  • Counselling
  • Life Coaching
  • Note taking
  • Medical smiles
  • Medical Reporting
• Virtual laboritys simulations
• Chain of thoughts methods
• One shot / Multi shot prompting tasks