ZySec-7B / README.md
leaderboard-pr-bot's picture
Adding Evaluation Results
55cf399 verified
metadata
license: apache-2.0
library_name: transformers
tags:
  - security
  - cybersecwithai
  - threat
  - vulnerability
  - infosec
  - attackio
  - cyber security
  - ai4security
  - llmsecurity
  - cyber
  - cybersec
  - cybersecurity
model-index:
  - name: ZySec-7B-v1
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 63.48
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aihub-app/ZySec-7B-v1
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 85.01
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aihub-app/ZySec-7B-v1
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 60.14
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aihub-app/ZySec-7B-v1
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 56.49
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aihub-app/ZySec-7B-v1
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 78.14
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aihub-app/ZySec-7B-v1
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 23.2
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aihub-app/ZySec-7B-v1
          name: Open LLM Leaderboard

ZySec-v1-7B

ZySec-v1-7B, stands as a pivotal innovation for security professionals, leveraging the advanced capabilities of HuggingFace's Zephyr language model series. This AI model is crafted to be an omnipresent cybersecurity ally, offering on-demand, expert guidance in cybersecurity issues. Picture ZySec-7B as an ever-present digital teammate, adept at navigating the complexities of security challenges.

The efficacy of ZySec-7B lies in its comprehensive training across numerous cybersecurity fields, providing a deep and wide-ranging understanding of the sector. ZySec is developed using the DPO technique, utilizing a varied dataset encompassing critical topics such as:

  • Sophisticated areas like Attack Surface Threats, Cloud Security, and the Cyber Kill Chain.
  • Key compliance and regulatory frameworks, including CIS Controls, FedRAMP, PCI DSS, and ISO/IEC 27001.
  • Practical aspects like Cloud Secure Migration, Data Exfiltration Techniques, and Security Incident Handling.
  • Crucial strategic fields such as Security Governance, Risk Management, and Security Architecture Review.

ZySec-7B's training spans over 30 unique domains, each enriched with thousands of data points, delivering unparalleled expertise.

As the first of its kind in an open-source, AI-driven cybersecurity series, ZySec-7B transcends the conventional role of a support tool, redefining organizational security approaches. Its open-source nature not only invites community contributions but also enhances its flexibility and transparency in managing vast cybersecurity data. ZySec-7B is instrumental in providing vital, actionable insights for strategic decision-making and advanced risk management. More than a mere software, ZySec-7B is a community-enhanced strategic tool, equipping your team to proactively confront and stay ahead of the dynamic landscape of cyber threats and regulatory demands.

Dataset Distribution

Details of dataset distribution here - Dataset Distribution

Fully compatible with LM Studio. Search for “Zysec” and here is what you get. Here is a sample output of ZySec writing email to John about database security using LM Studio:

Sample Output

The training is funded by AttackIO, the mobile app for Cyber Security professionals.

Official GGUF version is hosted here - ZySec-7B-v1-GGUF on HuggingFace

ZySec AI: Unleashing the Potential of the ZySec Series Model

Project ZySec, an integral part of ZySec AI, stands at the forefront of integrating Artificial Intelligence into Cybersecurity. Centered around the innovative ZySec 7B model, it's designed to revolutionize the cybersecurity landscape with AI-driven solutions. ZySec AI isn't just a tool, it's a transformative approach, blending AI's cutting-edge capabilities with the unique intricacies of cybersecurity, while ensuring privacy and security.

Discover the Key Features of Project ZySec

  • AI-Driven Cybersecurity: Tap into the power of the ZySec 7B model, a bespoke AI solution fine-tuned for cybersecurity.
  • 24/7 Expert Assistance: Benefit from round-the-clock support and expert advice, guaranteeing smooth operations during any SOC shift.
  • Efficient Playbook Access: Streamline your workflow with quick and easy access to playbooks and documents, enhancing information retrieval.
  • Standards Explorer: Navigate various standards with ease, akin to a seasoned expert's proficiency.
  • Ongoing Internet Research: Leverage AI-enabled, thorough internet research for exhaustive insights. (Note: Internet use is optional and specific to this feature).

About Project ZySec by ZySec AI

ZySec AI an opensource project, with a vision towards fusioning of Cybersecurity with Artificial Intelligence. Our goal is to transform the way security professionals engage with technology. More than a mere tool, ZySec AI symbolizes a comprehensive strategy to augment security operations, merging the innovative essence of AI with cybersecurity's distinctive challenges, always prioritizing privacy and security.

https://github.com/ZySec-AI/ZySec

The training is funded by AttackIO, the mobile app for Cyber Security professionals.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 61.08
AI2 Reasoning Challenge (25-Shot) 63.48
HellaSwag (10-Shot) 85.01
MMLU (5-Shot) 60.14
TruthfulQA (0-shot) 56.49
Winogrande (5-shot) 78.14
GSM8k (5-shot) 23.20