metadata

license: mit
library_name: transformers
datasets:
  - Severian/Internal-Knowledge-Map
pipeline_tag: text-generation

This model is the MLX trained version on the experimental 'Internal Knowledge Map' dataset. Training was done 100% local on my M2 Ultra 128GB. I've found that there are noticable differences between the Transformers (Unsloth) version I trained in a Colab and the ones I have been training locally using MLX. I personally prefer the MLX ones as they seem to train MUCH better and retain more of the aspects from fine-tuning. In particular, I also ripped this model to shreds and it still seems to work amazingly. Here was my training set up:

    model.freeze()
    for l in model.model.layers:
        l.self_attn.q_proj = LoRALinear.from_linear(
            l.self_attn.q_proj, r=128, lora_alpha=256, lora_dropout=0.001
        )
        l.self_attn.v_proj = LoRALinear.from_linear(
            l.self_attn.v_proj, r=128, lora_alpha=256, lora_dropout=0.001
        )
        l.self_attn.o_proj = LoRALinear.from_linear(
            l.self_attn.o_proj, r=128, lora_alpha=256, lora_dropout=0.001
        )

        l.self_attn.k_proj = LoRALinear.from_linear(
            l.self_attn.k_proj, r=128, lora_alpha=256
        )

        if hasattr(l, "block_sparse_moe"):
            l.block_sparse_moe.gate = LoRALinear.from_linear(
                l.block_sparse_moe.gate, r=16, lora_alpha=32, lora_dropout=0.001
            )

    p = sum(v.size for _, v in tree_flatten(model.parameters())) / 10**6
    print(f"Total parameters {p:.3f}M")
    p = sum(v.size for _, v in tree_flatten(model.trainable_parameters())) / 10**6
    print(f"Trainable parameters {p:.3f}M")
   

    trainingArgs = TrainingArgs(
        batch_size=4,
        iters=4200,
        val_batches=25,
        steps_per_report=10,
        steps_per_eval=200,
        steps_per_save=100,
        adapter_file="adapters.npz",
        max_seq_length=4096,
    )

Test this out and see if you find anything interesting or intriguing. I will keep iterating more versions but this one seems like a fun and useful way to start.

Phased Training Methodology

Leveraging this dataset, we've adopted a phased training methodology that focuses sequentially on different dataset components, namely "System" and "Instruction," across separate training phases. This approach allows models to build layered understandings from general systemic insights to specific instructional cues, enriching their generative output with both broad contextual awareness and detailed, topic-specific knowledge.

Phase 1: System Focus In the initial phase, the model concentrates on the "System" component, absorbing overarching guidelines and objectives. This phase lays the foundational understanding, enabling the model to grasp the contextual framework and systemic knowledge encapsulated in the dataset.

Phase 2: Instruction Focus Building upon the systemic knowledge, the second phase shifts the model's focus to the "Instructions" component. This phase sharpens the model's ability to interpret specific prompts and generate responses that are not only informed by the broader context but also precisely tailored to the instructional cues.

GGUF Q8 Version: https://huggingface.co/Severian/Nexus-IKM-Mistral-7B-GGUF

If you'd like to train your own version, here is the full notebook to recreate the training on Unsloth yourself (https://colab.research.google.com/drive/1828t77iO2nLRXVfB8HoI11eFu-79-Oe7?usp=sharing). You'll just have to drop in the train.jsonl from the Dataset repo (https://huggingface.co/datasets/Severian/Internal-Knowledge-Map) into your Colab directory and rename it dataset.jsonl

Example Outputs vs Base model

How can everyday household items be repurposed to solve common gardening problems?

Base Mistral-Instruct-v0.2:

There are several ways everyday household items can be repurposed to solve common gardening problems. Here are some ideas:

Plastic Bottles: Cut off the top of a plastic bottle and bury it with the bottom in the ground, upside down. This will act as a water reservoir for your plants, helping them stay hydrated during dry spells.
Egg Cartons: Use egg cartons to start seedlings indoors. Once the seedlings have grown large enough, you can transplant them directly into the ground, carton and all. The carton will decompose over time, leaving the seedling's roots undisturbed during the transfer.
Coffee Grounds: Use coffee grounds as a fertilizer for acid-loving plants such as azaleas, rhododendrons, and hydrangeas. Coffee grounds also help deter pests like slugs and snails.
Newspaper: Use newspaper to suppress weeds. Lay down several layers of newspaper over the weeded area, then cover it with a layer of mulch or compost. Over time, the newspaper will decompose and the weeds will not be able to grow through.
Baking Soda: Use baking soda as a fungicide for roses and other plants susceptible to powdery mildew. Mix 1 tablespoon of baking soda with 1 quart of water, then spray the solution onto the infected areas. Be sure not to overuse, as too much baking soda can harm the plant.
Empty Cans: Puncture the bottoms of empty tin or aluminum cans and bury them in the garden. The cans will act as self-watering planters for herbs or small vegetables like tomatoes. Fill the cans with soil, plant your seeds, then water the can from the top to allow the water to seep out through the bottom and slowly irrigate the roots.
Glass Jars: Use glass jars to create DIY terrariums. Fill the jar halfway with pebbles or small stones for drainage, then add a layer of soil and your favorite plants. The sealed jar will create a mini greenhouse effect, allowing the plants to thrive in humid conditions.
Old Tires: Repurpose old tires into raised garden beds by filling them with soil and compost. This is an excellent solution for creating additional growing space without breaking the bank. Be sure to drill several holes in the tire to allow water to drain properly.

Nexus-7B:

Repurposing everyday household items for gardening can be an innovative and cost-effective solution to common gardening problems. Here are some examples of how everyday items can be transformed into useful gardening tools or solutions:

Plastic Bottles as Waterers: Cut the bottom off a plastic bottle, poke several small holes in the sides for drainage, and turn it upside down to water your plants. This can be used to water hanging baskets, potted plants, or even in-ground flowers without overwatering them.
CDs as Sun Deterrents: Place old CDs around the perimeter of your garden to reflect sunlight and discourage birds from feeding on ripe fruits or vegetables. The bright reflections can also serve as an aesthetic enhancement for your garden.
Old Tires as Planters: Stack tires in a sunny location, fill them with soil, and add plants. This repurposed tire planter is ideal for growing larger plants such as tomatoes, cucumbers, or squash. Ensure that the tires are stable and secure before filling with soil.
Newspapers as Weed Blockers: Cover the ground around young seedlings or plants with old newspapers to prevent weed growth. This method is particularly useful in gardens where manual weeding is not practical due to the size of the area or physical limitations.
Egg Cartons as Seed Starters: Fill each cup of an egg carton with soil and plant seeds, watering them gently. Once the seedlings are large enough, you can transfer the entire carton into a larger pot. This method allows you to start seeds indoors using minimal space and resources.
Tin Cans as Seedling Protectors: Cut the bottom off an empty tin can, paint it with non-toxic, weather-resistant paint, and poke several drainage holes in the sides. Once painted and sealed, place the can over young seedlings to protect them from frost or excessive sunlight.
Toilet Paper Rolls as Seedling Markers: Write plant names on toilet paper rolls with a waterproof marker, then stick them into the soil next to their corresponding seedlings. This method helps keep track of plants during early growth stages.
Coffee Grounds as Fertilizer: Spread coffee grounds around acid-loving plants such as blueberries and tomatoes to add nitrogen to the soil. The addition of coffee grounds also improves soil structure and deters pests.
Old Pots as Drainage Hacks: Drill holes in the bottoms of old plastic or ceramic pots and bury them in the ground around larger plants or trees. These buried pots will serve as water reservoirs, providing additional moisture to the roots when natural rainfall is scarce.
Kitchen Waste as Compost: Collect fruit and vegetable waste, along with coffee grounds and eggshells, in a designated compost bin. Decompose these items over time to create nutrient-rich soil for gardening, saving money on commercial fertilizers.

Evaluation

Depth and Specificity

Base Model: Provides straightforward, commonly known uses for household items in gardening, like using plastic bottles for water reservoirs or egg cartons for seedlings. It covers a broad range of solutions but with a focus on practical, direct applications.
Nexus-7B: Delves into more innovative and less commonly known uses, such as using CDs as sun deterrents or tin cans as seedling protectors. This response demonstrates a deeper understanding and creative thinking in repurposing items for gardening challenges.

Integration of Knowledge

Base Model: Lists uses in a manner that aligns with general knowledge, lacking in the integration of broader environmental or ecological considerations.
Nexus-7B: Shows an integrated approach, suggesting uses that consider not just the gardening problem at hand but also potential aesthetic enhancements and ecological benefits, such as using coffee grounds as fertilizer for its nitrogen content and soil structure improvement.

Innovation and Creativity

Base Model: Sticks to more traditional and widely recommended practices. While useful, these solutions do not push the boundary of creative repurposing.
Nexus-7B: Exhibits a higher degree of creativity, especially in solutions like using CDs for bird deterrence or creating a mini-greenhouse effect with glass jars. These suggestions indicate a leap in applying lateral thinking to gardening problems.

Environmental and Ecological Consideration

Nexus-7B: Implicitly suggests a greater awareness of environmental impact by promoting composting and the reuse of materials in ways that benefit the garden's ecosystem, showing an undercurrent of sustainability that's not as pronounced in the base model's response.

How to Spot Nuances:

Look for Creativity: The Nexus-7B model's responses are more creative and less conventional, suggesting a deeper dive into each item's potential uses.
Consider the Depth of Integration: Nexus-7B integrates broader environmental and sustainability considerations, moving beyond the immediate utility to consider the impact on the garden's ecosystem.
Assess the Innovation Level: Responses that suggest new or unconventional uses demonstrate the model's ability to synthesize information creatively and offer innovative solutions.
Evaluate Environmental Consciousness: Nexus-7B's approach reflects a subtle yet significant emphasis on sustainability, indicating an advanced integration of ecological awareness into problem-solving strategies.

Most people recognize bees for their role in pollination, but their contributions to ecosystems extend far beyond that. Can you elaborate on the less commonly known yet crucial roles that bees play in supporting biodiversity and ecosystem health?

Base:

Absolutely! Bees are essential pollinators, but they contribute significantly to various aspects of ecosystem health and biodiversity in ways that are not always recognized. Here are some less commonly known roles bees play in supporting ecosystems:

Nectar and pollen collectors: Bees collect nectar and pollen from flowers to feed themselves and their hives. In doing so, they help transport nutrients from the plants to other organisms, creating a complex web of connections between different species. This not only supports the growth and survival of plants but also ensures food sources for various animals that rely on those plants.
Decomposers: Bees play a role in decomposition by regurgitating undigested parts of flowers, which are rich in pollen and nectar, as food for their larvae. This process helps break down plant material and recycle nutrients back into the soil, contributing to soil health and fertility.
Pest control: Bees contribute to pest control by attracting various natural enemies of insects that damage crops or other plants. For example, bees help pollinate flowers that produce parasitic wasps, which in turn prey on harmful aphids and other pests. By supporting the populations of beneficial insects, bees indirectly protect crops and other vegetation from damaging pests.
Seed dispersal: Some bees play a role in seed dispersal by unintentionally carrying seeds on their bodies as they move between flowers. This helps ensure the distribution and growth of various plant species and contributes to ecosystem diversity.
Honey production: Bees produce honey, which has numerous uses for humans beyond just consuming it as food. Honey has antibacterial properties and can be used as a natural sweetener in cooking or as an ingredient in various medicinal products. In addition, the production of honey creates economic opportunities for beekeepers and supports local communities.
Nutrient cycling: Bees contribute to nutrient cycling by transferring pollen from one flower to another during the pollination process. This helps ensure that plants are able to produce seeds, which in turn allows for new plant growth and contributes to ecosystem health and productivity.
Carbon sequestration: Bees play a role in carbon sequestration through their interactions with various plant species. By facilitating pollination and supporting the growth of vegetation, bees contribute to the absorption and storage of carbon dioxide from the atmosphere. This helps mitigate the effects of climate change and supports overall ecosystem health.
Supporting food webs: Bees contribute to complex food webs by providing essential resources for various organisms. For example, bee larvae serve as food sources for birds and other predators, while adult bees provide nectar and pollen for a wide variety of insects and other animals. This helps maintain the balance and diversity of various ecosystems.

Nexus:

Absolutely! While bees are primarily known for their role in pollination, their contributions to ecosystems encompass a much broader range of functions that support biodiversity and ecological health. Here are some of the less commonly recognized yet crucial roles bees play:

Seed Dispersal: Bees not only transfer pollen between flowers but also unintentionally pick up and distribute seeds as they forage, aiding in the dispersal of plant species. This is particularly important for plants that have small seeds or rely on insects for seed dispersal.
Nitrogen Fixation: Some bee species, like leafcutter bees, play a role in nitrogen fixation by collecting plant material and transporting it back to their nests. This process improves soil health, benefiting the growth of nearby plants.
Soil Aeration: Bees contribute to soil aeration through their foraging activities. As they excavate burrows for nesting or collect soil for building cells in their nests, they create small tunnels that improve soil structure and promote water retention.
Food Source for Other Organisms: Beyond pollination, bees provide food for a wide range of organisms, including birds, mammals, and other insects. This makes them crucial components of complex food webs within ecosystems.
Biological Pest Control: Bees also contribute to biocontrol by preying on pests that threaten crops or natural ecosystems. For example, parasitic bees and wasps lay their eggs in the larvae of various pests, thereby reducing their populations.
Medicinal and Pharmaceutical Uses: Bees play a significant role in producing honey and beeswax, which have medicinal and pharmaceutical applications. For instance, honey has antibacterial properties that can help treat wounds, and beeswax is used as a base for cosmetics and in the manufacturing of candles.
Biodiversity Indicators: Bees serve as valuable indicators of biodiversity health. Their distribution and abundance reflect the overall health of ecosystems and can signal changes in environmental conditions that may affect other species within their habitats.
Cultural and Spiritual Significance: Bees have cultural and spiritual significance in many societies around the world. They are celebrated for their role in agriculture and their contribution to honey production, which is used in various traditional practices and religious ceremonies.

In summary, bees contribute significantly to ecosystems beyond pollination by enhancing soil health, supporting food webs, providing resources for medicinal and pharmaceutical applications, serving as indicators of biodiversity, and playing cultural and spiritual roles within communities. These diverse functions highlight the crucial role bees play in maintaining ecological balance and promoting sustainable agriculture.

Evaluation

Depth of Information

Base Model: Offers a broad overview of bees' roles, touching on common but critical functions like nutrient cycling, honey production, and seed dispersal. It provides general explanations that align with widely known facts about bees.
Nexus-7B: Delves deeper into specific and less commonly recognized contributions of bees, such as their role in nitrogen fixation and soil aeration, showcasing a nuanced understanding of ecological processes.

Breadth and Specificity of Contributions

Base Model: Lists a variety of bees' ecological roles with a focus on their immediate environmental impacts, such as decomposers and participants in pest control. It effectively communicates bees' contributions but within a relatively conventional scope.
Nexus-7B: Expands the conversation to include specific ecological functions, such as bees' contribution to soil aeration and their significance in biocontrol, presenting a more comprehensive picture of bees’ ecological roles. This model's response implies a deeper integration of ecological knowledge, highlighting less obvious but crucial ecosystem services provided by bees.

Innovative and Unique Insights

Base Model: While informative, the response sticks to more traditional explanations of bees' ecological roles without introducing novel insights or making connections to broader ecological or societal impacts.
Nexus-7B: Introduces innovative aspects of bees’ ecological contributions, like their role in nitrogen fixation and as indicators of biodiversity health, providing insights that reflect a layered understanding of ecosystems. It also notes the cultural and spiritual significance of bees, bridging ecological roles with societal impacts.

How to Spot Nuances and Understand Impact

Depth and Detail: Look for responses that provide detailed explanations of processes and mechanisms, like Nexus-7B’s mention of nitrogen fixation, which indicates a deeper understanding of the subject matter.
Breadth of Ecological Roles: Assess the range of ecological roles discussed. Nexus-7B’s mention of soil aeration and biological pest control shows a broader perspective on bees’ environmental impact.
Integration of Knowledge: Notice how different pieces of information are woven together to paint a comprehensive picture. Nexus-7B’s response integrates bees' ecological functions with their importance in cultural and spiritual contexts, demonstrating an advanced synthesis of information.
Uncommon Insights: Pay attention to unique or less commonly discussed roles that bees play, as highlighted by Nexus-7B. These insights can reveal the model's ability to go beyond surface-level knowledge and explore complex ecological dynamics.