Nekochu/Luminia-13B-v2-QLora

Same card and base of LoRA Luminia-13B-v1-QLora continue on Pre-Training Dataset bigcode/starcoderdata, this is meant to be a experimental.

Stage a - 0.22 epoch:22day (660500 steps) on

>

starcoder_zig,starcoder_visual-basic,starcoder_standard-ml,starcoder_common-lisp,starcoder_cuda,starcoder_elm,starcoder_stata,starcoder_coffeescript,starcoder_vhdl,starcoder_clojure,starcoder_cmake,starcoder_systemverilog,starcoder_dart,starcoder_dockerfile,starcoder_css,starcoder_emacs-lisp,starcoder_erlang,starcoder_f-sharp,starcoder_julia,starcoder_yaml,starcoder_fortran,starcoder_python,starcoder_ada,starcoder_agda,starcoder_alloy,starcoder_antlr,starcoder_applescript,starcoder_assembly,starcoder_augeas,starcoder_awk,starcoder_batchfile,starcoder_bluespec,starcoder_r,starcoder_racket,starcoder_restructuredtext,starcoder_rmarkdown,starcoder_ruby,starcoder_rust,starcoder_sas,starcoder_scala,starcoder_scheme,starcoder_shell,starcoder_smalltalk,starcoder_solidity,starcoder_sparql,starcoder_sql,starcoder_stan,starcoder_jupyter-scripts-dedup-filtered,starcoder_jupyter-structured-clean-dedup,starcoder_kotlin,starcoder_lean,starcoder_literate-agda,starcoder_literate-coffeescript,starcoder_literate-haskell,starcoder_lua,starcoder_makefile,starcoder_maple,starcoder_markdown,starcoder_mathematica,starcoder_matlab,starcoder_ocaml,starcoder_yacc,starcoder_pascal,starcoder_perl,starcoder_php,starcoder_powershell,starcoder_prolog,starcoder_protocol-buffer,starcoder_git-commits-cleaned,starcoder_github-issues-filtered-structured,starcoder_glsl,starcoder_go,starcoder_groovy,starcoder_haskell,starcoder_html,starcoder_idris,starcoder_isabelle,starcoder_java-server-pages,starcoder_java,starcoder_javascript,starcoder_json,starcoder_tcl,starcoder_tcsh,starcoder_tex,starcoder_thrift,starcoder_typescript dataset_info.json

Stage b - 1 epoch:306h (235000 steps) on `starcoder_c-sharp,starcoder_c,starcoder_cpp,starcoder_elixir,starcoder_verilog,starcoder_xslt`

Note: seems like 1GB LoRA got a catastrophic forgetting given that's TB dataset... Stage B has no knowledged of Stage A