Modernizing Government Security
Operations Centers With A
Cybersecurity Lakehouse

Government IT leaders understand the importance of a strong cybersecurity posture, but they’ve been stuck fighting modern adversaries withoutdated tools.
The urgency to tackle that challenge has intensified in recent years given the July 2023 release of the National Cybersecurity Strategy
Implementation Plan that furthers the prior Executive Order on cybersecurity and the M-21-31 memorandum.
But it’s not just a matter of executive mandate. Multiple high-profile security breaches have shone a spotlight on the need for federal agencies
to modernize Security Operations Centers (SOCs) to ensure their cyber posture can adequately protect valuable data and other assets against
a dynamic array of threats. The SolarWinds hack, Microsoft IIS exploit, and Log4j vulnerability made it crystal clear that legacy tools andprocesses are not up to the task of protecting valuable agency systems and data.

Legacy security tools
can’t keep up

 odernize with a Cybersecurity
M
Lakehouse from Databricks

These hacks — coupled with the Executive Order — reveal

The Databricks Lakehouse Platform is a combination ofthat a 90-day audit trail is insufficient, especially given thethe best elements of data lakes, data warehouses, data

complexity of modern IT infrastructure and software supply

engineering and data science/ML in a secure, reliable, andchains. Federal IT teams must operate with a mindset thatcompliant solution. The Cybersecurity Lakehouse approachassumes breach — and that gives them the proper tools andcan collect and ingest any data into a single platform forprocesses to detect, mitigate and audit security incidents

secure data pipelines, data normalization, incident response,

whenever and wherever they occur.

threat hunting and advanced analytics — in a scalable

Legacy SIEM tools and architectures, however, weren’t
designed for this modern paradigm. In particular, they’re notgood at separating compute from storage — a necessity
for teams that want to be able to look back a year (or more)at their security data. They also struggle with modern
federation requirements, and typically lack the advanced
analytics capabilities including AI/ML that more and more IT

manner that can support retention and audit periods of morethan one year.
Highlights of the the platform include:•True multicloud support•

HiddenLayer for thwarting adversarial attacks on
AI/ML models

teams want.

•Large Language Model (LLM) explainability — no black boxes•

Zero trust architectural support

Yet budget and other operational constraints means federal•

Advanced data science & analytics capabilities

CIOs can’t simply replace and modernize their SOC overnight.

•A robust partner ecosystem ready to support agencies’

Fortunately, there’s a way for agencies to modernize now
while also creating a migration path to next generation
security architecture when they’re ready.

specific needs

Databricks for SOC Modernization
SourceIngestTransform

Query / Process

ServingAnalyze

Lakehouse Platform

Sensors and IoT(unstructured)
Event
Streaming

Unity Catalog

SOAR

RDBMS
(structured)
Structured
Streaming

SIEM
Data
Pipelines

Media(unstructured)

Detections

ThreatHunting

AdvancedAnalytics

3rd Party

Databricks Compute
Files / Logs
(unstructured)
Compute is separated from
customer’s cloud storage

Business Apps
(structured)

Ingest Tool

Cloud Storage

Raw

Other Clouds

The Lakehouse approach also gives your agency theflexibility and scalability required to modernize on yourschedule. Since the Databricks Lakehouse is fundamentally a•

Build a migration bridge from legacy to modernization:
Take a phased approach to modernization that doesn’t
require ripping-and-replacing your existing SIEM and— on your timeframe — while simultaneously enhancing andother SOC tooling•This approach lets you migrate from legacy to modern in a

Dramatically lower TCO: Support high-velocity data growthneeds for ephemeral compute and cloud object storage —very stage-and-phase manner. Over time, the Cybersecurity
Lakehouse becomes your modern data platform and modern

Dimensional

Cybersecurity Lakehouse advantages include:

data integration platform, you can ingest data incrementally
modernizing your SOC and cyber posture.

Enhanced

at agency level scale — while actually lowering costs•

Gain flexibility in architecture: Keep sensitive security

integration platform, creating the foundation for modern

data within your agency’s own cloud account in an open

tooling and architectures.

format that facilitates integration with various SIEMs andSecurity Operations tools

Advantages of the Cyber­security Lakehouse approach
The overarching benefit for agencies is the ability to•

Extend and enhance search retention periods: Monitor datawith long retention periods at agency data scale (Petabyte+),without additional storage architecture or significant costs•

Add AI security analytics: Implement advanced analytics
today’s federal IT pros need, such as anomaly detection

respond to growing pressure to boost your cybersecurity,using Databricks’ leading data science and machineon a timeline that is actually attainable. It’s a cost-effective

learning platform

method of meeting recent cybersecurity directives while
also enabling future modernization.

•Ensure explainability: Eliminate data silos black box
security algorithms

Meeting M2131: Paving the road fromlegacy tools to the modern SOC
The recent executive directives on cybersecurity aren’t

As MITRE notes in its “11 Strategies of a World-Class Cyber­surprising — they’re necessary. The modern threat landscapesecurity Operations Center,” the practice of running a modernis too complex and dynamic for legacy architectures anddata & ML platform alongside a legacy SIEM has becometools. But there’s no time-warp that can take you from legacy

popular enough that it has a name: the “dual stack” approach.

to super-modern overnight. Your IT teams must work within
their operational mission — and their operational challenges— to take a phased approach that stays within budget.

Think of it as a best-of-both-worlds strategy, one thatempowers agencies to meet recent directives andmodernize their security operations in a pragmatic manner.

Building a Cybersecurity Lakehouse on the Databricks

That’s the Databricks Lakehouse Platform.

platform allows exactly that: A cost-effective, flexible andscalable approach to SOC modernization that can work
alongside your existing SIEM and other tools.

Sources

Asset Knowledge

Event-Based Data

Cyber Threat Intelligence• Business Information• IoC Feeds• Patch and Conﬁg Status• Incoming Tips from Other SOCs

• Risk Scoring• Finished Reporting• Firewalls & Proxies• Applications

Data Enrichment• IoT/ICS/SCADA

Pivot to Context• Mobile• Cloud

Event Ingest

Event Ingest

Host-Based Sensors• Anti Virus• Firewall• Endpoint Detection & Response• Indicator Sweeping

Event Message Bus• Domain Controller & OS

Threat Intel Management
Data Enrichment• Finished Intel Repository

Event Ingest• Adversary & Campaign Knowledgebase
Pivot to Context• Indicator Curation• Remote Imaging/Forensics
Event Ingest

Network-Based Sensors• Custom Detection NIDS/NIPS• Trafﬁc Metadata & Proﬁling• File Carving

Pivot to Analysis• Extraction of Files Seen “On the Wire”

AI/ML Enrichment

SOAR WorkﬂowManagement• Ticketing

Case Escalation• Escalation• Metrics & KPIs

• Static Analysis• Automation & Orchestration• Dynamic Runtime Analysis• Reporting

Original Source: Mitre, 11 Strategies of a World-Class Cybersecurity Operations Center

Learn more
Ready to start your SOC modernization journey?
Learn more at databricks.com/cybersecurity-analytics.

© Databricks 2023. All rights reserved.

Pivot to Context

Malware Detonation& File Analysis• NetFlow• Ad Hoc Trafﬁc Capture & Reconstruction

Alert Ingest