Isolation and Multi Level
Security on Lustre
DataDirect Networks Japan, Inc.
2016/10/18

Shuichi Ihara (sihara@ddn.com)© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

2

Multi Level Security

http://www.acq.osd.mil/ecp/Articles/FP_2016FEB.html© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com3Security is important for HPC Storage▶ Today, HPC storage is NOT just scratch anduser home directory use case is commonplace• Same cluster with various use cases• Dedicated hardware not efficient• Secured data accessible/visible ONLY to people who havecredentials and are authorized© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

4DDN contributions on Lustre Security andResource Management▶ Integration with Kerberos Authentication (Lustre-2.7)▶ Subdirectory Mount (Lustre-2.9)▶ Quota for Subdirectory (Lustre-2.10)▶ Docker Integration• With Subdirectory mount (Lustre-2.9)• With Kerberos Authentication (Lustre-2.10)• QoS for Docker container(Lustre-2.10+)▶ MLS (Multi Level Security) for Lustre© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com5What is Lustre Isolation?
▶ Lustre Isolation:• Provides isolated namespaces from a single filesystem▶ Lustre Isolation combines features of:• Containerso Each container mounts Lustre as a cliento ‘root’ user is allowed inside containers• Kerberoso Each container authenticates with its own credentials• Subdirectory mounto Each container is allowed to mount only a portion of thenamespaceo Allowance depends on client’s credentials© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com6

Lustre: Isolation
1. credentials(A)Kerberos server

2. credA
nodemap

Container A
mount(credA)
Container B/filesetA
Lustre client

Lustre servers© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com7What benefits from Lustre Isolation?
▶ Containers avoid static distribution of client nodes=> dynamic container images instantiation• No need to dedicate groups of clients to each population• Every client is available for any population• Several populations can share same client nodes at the sametime▶ Lustre Isolation enables:• Different populations of users on the same file systems• Isolation of these different populations of users
Þ Isolation makes Lustre multi-tenant© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

8

Taking Lustre Isolation a step further▶ Ability to isolate users from the same population• Prevent users from accessing others’ data
• Flexibly adjust access capabilities• But still share the same file system root
▶ Container doesn't know other container'ssecurity level• No associated with each container's security levelÞ Use SELinux MLS to enforce data confidentiality© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

9SELinux support on Lustre client side
▶ We already have Targeted policy support!

• Initial landing in 2.8• Optimizations available in 2.9+
▶ Now we need to support MLS on Lustre client© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

10

SELinux concepts▶ Targeted policy• Targeted policy defines confined and unconfined domains
for processes and users.
• It requiers to store security information permanently in fileextended attribute, to remember security context inheritedfrom the user and process that created the file.

© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com11SELinux concepts▶ Multi-Level-Security (MLS) policy• Adds the concept of security levels in addition to domains© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

12

SELinux concepts▶ Difference between targeted and MLS policies:• Targeted policy protects the OS• MLS policy protects the data▶ From a file system perspective• MLS works on clients like Targeted policy
o Use of security.selinux xattr to store security context

system_u:object_r:default_t:s2:c17© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com13SELinux concepts▶ Distributed file systems specificity:• Really need to make sure data is always accessed bynodes with SELinux MLS policy enforcedo Otherwise data is not protectedÞ Make sure SELinux cannot be disabled by root• secure_mode_policyload SELinux boolean
Þ Check SELinux status on client© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com14SELinux status on client▶ Enable SELinux!
▶ We need to make sure:• SELinux is enforcedo /sys/fs/selinux/enforce• The right policy module is loaded
o /etc/selinux/config• The policy is not alteredo Binary representation of policy at:/etc/selinux/<name>/policy/policy.xx© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

15SELinux status on client▶

Build “SELinux status” info
• With new usermode helper ‘l_getsepol’o because need to read and parse fileso because no SELinux API available in kernel to get this info

• Called from Lustre client code
• “SELinux status” info in the form:<1-digit enforcement>:<policy name>:<policy checksum>• Write “SELinux status” info to/proc/fs/lustre/<obd type>/<obd name>/srpc_sepol© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

16

SELinux status on client▶ SELinux status must be checked:• At connect time
• Every time the client accesses Lustre namespaceo openo createo unlinko rename• Every time the client might access security context
o getxattr
o setxattr

Þ add “SELinux status” info to these requests© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com17SELinux status on client▶ On Lustre server’s side• store “SELinux status” reference informationo in new ‘sepol’ field of nodemaps– can be different for different groups of nodes• compare “SELinux status” info received from client with‘sepol’ stored in nodemap
o match => process request normallyo no match => return Permission Denied (EACCES)© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com18MLS on Lustre client▶ What about performance?

• R&D test-bedo Environment– 1 client node, 1 server for Lustre MDS, OSS embedded in SFA 14KEo Hardware– Client node
» 16 cores» 128 GB RAM» IB 4X FDR– MDS node
» 48 cores» 128 GB RAM» RAID 6 10 x 900GB
10K SASo Software– CentOS 7.2 (3.10 kernel)– Lustre master (2.8.57)– MOFED 3.3• Objectiveo impact over metadata performance© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

19

MLS on Lustre client© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

20

MLS on Lustre client© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

21

MLS on Lustre client© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com22

Lustre MLS support – code status▶ Work in progress• Needs further optimizations• Code cleanup▶ Will push to Community when done
▶ Interested in early evaluation?

Please contact us!

© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com23

Lustre enhanced Isolation – use case
▶ Customer requirement to deliver "science as aservice” to:• internal groups• external commercial customers▶ Typical workload represented by the cgpbox

project• encapsulates the core Cancer Genome Project analysis
pipeline in a Docker image• https://github.com/cancerit/cgpbox© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com24

Lustre enhanced Isolation – use case▶ ‘cgp’ population only sees datastore

subdirectory▶ /datastore/input• Needs to be readable for every member of the ‘cgp’population
Þ Set security context’s level of directory to s0▶ /datastore/output/<id>• Accessible read/write for members of the same teamÞ Run container with:--security-opt label:level:s1:cxxx© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com25Summary▶ We are able to enhance isolation feature for Lustre• leveraging SELinux MLS policy• controlling SELinux status at the Lustre level▶ MLS feature works in conjunctions with otherLustre security features• Kerberos authentication• Sub directory mount▶ Enabling SElinux on Lustre Sever• Strict network security level checking on server and client© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comThank You!
Keep in touch with us

Team-jpsales@ddn.com

102-0081
東京都千代田区四番町6-2
東急番町ビル 8F@ddn_limitless

TEL:03-3261-9101
FAX：03-3261-9140

company/datadirect-networks© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

27

MLS on Lustre client© 2016 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com