---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb
- PleIAs/YouTube-Commons
- allenai/WildChat-1M
- Salesforce/xlam-function-calling-60k
- ShareGPT4Video/ShareGPT4Video
- OpenGVLab/ShareGPT-4o
- TempoFunk/webvid-10M
- MBZUAI/VideoInstruct-100K
- Isaak-Carter/j.o.s.i.e.v4.0.1o
- NousResearch/dolma-v1_7-c4
- NousResearch/dolma-v1_7-cc_en_head
- nyu-visionx/Cambrian-10M
- LargeWorldModel/ultrachat_qa_mix_1M
- LargeWorldModel/ultrachat_qa_mix_512K
- LargeWorldModel/ultrachat_qa_mix_256K
- LargeWorldModel/ultrachat_qa_mix_128K
- nkp37/OpenVid-1M
language:
- de
- en
library_name: mlx
tags:
- moe
- multimodal
- vision
- audio
- endtoend
- j.o.s.i.e.
---

# J.O.S.I.E. (Just a Smart and Intelligent Entity)

Welcome to the J.O.S.I.E. project repository! J.O.S.I.E. is a cutting-edge, super intelligent AI assistant designed to revolutionize the way we interact with smart home systems and general AI capabilities. This document provides an overview of J.O.S.I.E.'s features, capabilities, and development roadmap.

## Table of Contents

1. [Introduction](#introduction)
2. [Features](#features)
3. [Training Stages](#training-stages)
4. [Current Progress](#current-progress)
5. [Usage](#usage)
6. [Contributing](#contributing)
7. [License](#license)

## Introduction

J.O.S.I.E. stands for "Just a Smart and Intelligent Entity." It is not just a conversational AI assistant but a fully multimodal AI designed to understand and process images, videos, thermal images, depth, and audio in real-time. J.O.S.I.E. is built to autonomously manage smart homes and provide general-purpose assistance, with advanced capabilities accessible only to the main user.

## Features

- **Real-Time Processing:** J.O.S.I.E. operates in real-time, ensuring quick and efficient responses.
- **Tool Calling:** Capable of calling various tools to perform tasks (only for the main user).
- **Short/Long-Term Memory:** Remembers past interactions and uses this data to provide a more personalized experience.
- **Secure Information Access:** Accesses top-secret information upon receiving a special password from the main user.
- **Contextual Greetings:** Greets users based on contextual data such as time of day, birthdays, and more.
- **Voice Interaction:** Will support real-time voice responses with a response time under 0.3 ms.
- **Advanced Multimodal Capabilities:** Initially uses Meta's image binding model, transitioning to a self-implemented encoder.
- **Uncensored Interaction:** Full, uncensored interaction capabilities are reserved for the main user.
- **Autonomous Smart Home Management:** Manages smart home devices and systems autonomously.

## Training Stages

J.O.S.I.E.'s development is structured into several meticulously planned stages, each focusing on different aspects of its capabilities:

### Stage 1: **Genesis**
- **Objective:** Fine-tune the Large Language Model (LLM) with a custom dataset and prompt format. The LLM used is Qwen2 7B and 0.5B.
- **Outcome:** A robust foundation for text-based interactions.

### Stage 2: **Fusion**
- **Objective:** Train encoders separately using transfer learning to align input embeddings with text embeddings.
- **Outcome:** Harmonized multimodal input processing.

### Stage 3: **Synergy**
- **Objective:** Fine-tune the LLM for multimodal reasoning using a custom dataset.
- **Outcome:** Enhanced reasoning capabilities across text and other modalities.

### Stage 4: **Vocalize**
- **Objective:** Fine-tune the decoder for audio output, giving J.O.S.I.E. a voice.
- **Outcome:** Synchronized text and audio responses.

### Stage 5: **Convergence**
- **Objective:** Perform full model fine-tuning for seamless integration of all components.
- **Outcome:** A fully multimodal, real-time interactive AI assistant.

## Current Progress

J.O.S.I.E. is currently in its beta stage, specifically in Stage 1. The model is being actively developed, and the current version is focused on fine-tuning the LLM with custom datasets.

### Latest Beta Version 4 of Stage 1:
- **Model:** [Isaak-Carter/josiev4o-7b-stage1-v0.1](https://huggingface.co/Isaak-Carter/J.O.S.I.E.v4o-7b-stage1-v0.1-gguf)
- **Quants:** [Isaak-Carter/J.O.S.I.E.v4o-7b-stage1-v0.1-gguf](https://huggingface.co/Isaak-Carter/J.O.S.I.E.v4o-7b-stage1-v0.1-gguf)

For a sneak peek at the current progress, visit the [GitHub Repo](https://github.com/Goekdeniz-Guelmez/J.O.S.I.E.-v4o.git).

## Source Code

To se the latest updates on J.O.S.I.E.v4o you can see my <a href="https://github.com/Goekdeniz-Guelmez/J.O.S.I.E.-v4o.git">Github Repo</a>
   
## Contributing

I welcome contributions from the you! To contribute to J.O.S.I.E., please fork the repository and create a pull request with your changes. Ensure that your code adheres to my coding standards and includes appropriate tests and comments.

## License

J.O.S.I.E. is licensed under the Apache2 License. See the [LICENSE](LICENSE) file for more details.