language:
- en
18/05/24 - Random Update
what's up, gang?
So, where have I been? Here and there. a few private testing runs on llama 3.
Opus-Instruct Single Turn is done. Multi-turn working. c2 Logs were a fun side-distraction.
my stacked merge projects, 8x22b tune, all that? was honestly held back because of llama 3. I really want to work on them but I'm procrastinating... a lot. i have pretty much the tools setup. just have no desire to run the training runs itself.
random life updates: I got promoted to LCP, aced my EMTCT tests, so that's cool? also if any Singaporean's here, you seen the Tampines deadly multi-car accident last month? yeah, I responded to that. was honestly a sobering sight. damn.
anyway, ya boy's still alive and active. and cooking. I'll try to release something by the end of this month.
18/04/24 - Where am I?
had a few messages asking if I'm alive. yeah, I am. I occasionally talk in the KoboldAI or SillyTavern discords. Anyway, life update. Had a short trip to Batam for Hari Raya and all that, it was a short and refreshing break. You know why Batam is popular;)
where are all the projects I promised? I'm... working on them. yeah. So many new cool things came out, including the Maxtral models, that seemed easier to tune on, just more expensive. Really, I'm working on them. ($75 per 1M output tokens for Claude Opus is outrageous kek, and I'm working on ~10M tokens) The stacked merge and finetune experiments? working on those too. For my old followers, Hesperus? shit, I'm remaking those.
09/04/24 - Off topic
being an EMT honestly is fun work? kind of stressful at times, and 12h shift sucks, but its fun? a peek into my life so far lol. writing it out is really how to say it, stress relieving? relaxing? cathartic?
some cases are honestly bad and stressful, theres the occasional trolling uncles, routine cases and everything. kept me at my toes the whole shift, especially when we were first call. look ma, I'm on the news?
totally routine topups that shift:
Penthrox x1
Tramadol x1
Nasal Cannula x2
Adult SFM x1
Adult NRM x1
Adult BVM x1
Adult Defid Pad x1
IV Admin Set x1
Ns 500 ml x1
20G Cannula x2
21g Cannula x1
3ml Syringe x1
Tegaderm x1
10cm Crepe Bandage x1
7.5cm Crepe Bandage x1
5cm Crepe Bandage x2
Gauze x1
Triangular Bandage x5
Large Pad x2
Small Pad x2
Surgical Mask x50
Linen x3
L Gloves x100
BP Sharp Tag x1
Ice pack x3
4/7
This shouldn't violate any privacy thing? there's like no details leaked, ok govt?
05/04/24 - Mini Updates
what's up huggers, it's like 5am here, on my phone, and I'm working night shifts. yeah boi.
fimb v3 stuff is slow as I mentioned on their page, if you wanna support me on ko-fi (on my page), that's cool. if not, i have my own money, feel free, support helps, appreciate those who do.
Why shit is slow:
- aphrodite-engine does not want to cooperate with me for dataset generation (toxic / nsfw instruct datasets)
- Claude Opus isn't on AWS yet, anthropic console sucks balls (I'm not American, sucks for me). I'm using Claude Sonnet in the mean time.
- I'm just so tired after shifts. plus, I'm getting called back again because someone else is sick so less time for me, have to help cover another rota. shame, planned on working on my data that day.
- Burnout, crash out? don't really know. i worked really hard on all this, I'm feeling the effects. I'm someone who well, burns themselves out after going hard and focusing 100% on a task. it's a habit I'm trying to fix.
have a good day, or night, wherever u are.
29/03/24 - Sup
Whats up gang its your favourite man out here.
Fimb v3 is in the works but loss values are fucked with latest axolotl versions? Unsure, but I know the seperate datasets trained fine in the past, now they're at > 13 loss in the beginning? Idk what's going on here.
Solstice v2 is in the works, dataset expanding.
Senko is in a rock right now, having issues with what I want it to do exactly. Test versions have their issues and all, but its not as much of a timesink as Fimbulvetr-v2 was.
Typhon and Glacier finetunes are on the backburner, lack of time mainly, plus its the fasting month and all. Not much energy after 12h night shifts.
Claude 3 instruct data being worked on and all.
The main hardest thing about finetuning and all is the data curation, training is just money-consuming but easy to monitor and manage. The data creation, filtering, cleaning? That's hard.
mmm, funds are there, just need to finish curating data to tune.
14/03/24 - Updates
I miss her man
anyway im doing some experiments so feel free to try out Shiki and Senko, or not up to you tbh
ymmv with the models they are kinda sensitive to prompting
and yes I'm not kinda sound right now mentally but we ball
ohio out
09/03/24 - Just rambling on about my future plans
I don't even know what to do man. So many ideas, so little money and time.
I'm practically waiting on my paycheck so I can continue training.
Current Plans
14B-Glacier-Finetune - Sure a Full Finetune would be good but that would be expensive. A LoRA maybe? The goal is to smoothen out the tensors and repair brain damage that happens after frankenmerge.
Typhon-Mixtral Finetune - I have compiled a nice dataset to further fine-tune on this merge of mine, to hopefully improve results. Unsure if I should further DPO this one too? On non-gpt4 data of course unlike current DPO pairs.
11B-Fimbulvetr-v3 - I have ideas. Unsure if it would make it that much better or if it will be a side-grade.
Claude-3 Output-based Instruct Dataset - Cool Idea maybe? It'll introduce atleast a different '-ism' compared to GPT-4 ones out there which are generic. Claude is somewhat more human-like. I do have access so this is not as big of an issue. I just need more time.
Yi Tunes - Yeah this is on the backburner for a while. I'll think on it. I've struggled with Yi-tunes in the past. Most of mine are privated.
Ignoring that, man having gauze packed into a wound hurts like hell without painkillers. My bad, old patients of mine back when I was a nursing student. You all made it look not that bad.
Anyway, here's something I've been listening to this past week. Some random music for someone who read this entire thing haha.
Today's Brainrot:
Initial Ramble:
Someone questioned my model naming scheme. Good question. Here's how I chose my model names.
Stheno --> 13B L2 Merge Serie ---> It all Began Here --> stolen from discord, someone's idea
Medusa --> 7B L2 Merges ---> DOA
Euryale --> 70B L2 Merge Series --> Gorgon Sisters Continuation
Zephyrus --> L1 Merge --> Sounded Cool tbh
WinterGoddess --> L2 70B Merge + Tune --> Training Run Name: (Autumn Royal, Winter something I forgot.) Decided to brainstorm some fun names. Yeah. Dont know where the Goddess part came from, but I was writing some ERP between a guy and a goddess, and the test model helped critique it, so yeah.
Why obsession over the theme of Winter? I do not know where to begin. I was sick, delirious during training break, and my brain got obsessed with the theme of winter. I stuck my head in the freezer once. I ate ice. I'm silly like that.
Fimbulvetr --> Final Winter. Represents the end of it all. My peak Solar model imo.
Frostwind --> Initally wanted something to do with Mistral + Solar, had Claude suggest Frostwind as a name. Initial plan was to train Mistral, but solar came out so I took that name instead.
Solstice --> Solar, Sun, Peak, or (Orgasm, you know?) since it was a Lewd/NSFW Instruct Dataset
Sensualize --> Had GPT4 generate me a name that is for an NSFW model. Yeah.
Older Model Names:
Nyaa --> Meow. Trained on cat assistant vibes.
Winterreise --> As above, training run for another model was named Sonata. sounded cool as fuck when i asked a model to merge Winter + Sonata.
Nyakura --> I was obsessed with cats during that time.
Venomia --> Trained over toxic datasets.
Ana -> 7B Mistral Merge + Tune --> Medusa revived, you know, that FGO Ana.
Lila --> RP with a character, obsessed with her while testing Euryale 1.5 (WinterGoddess)
Solus --> RP with a character, obsessed with her while testing Euryale 1.5 (WinterGoddess)... yeah.
Next question. What's the worst cases I've seen so far?
A traumatic arrest, a few cardiac arrests, accidents here and there. Most shifts are okay though. Some shifts are just tough but it is what it is.