Package: lvfs
L Virtual File SystemRationaleLVFS is a Python interface wrapping several storage APIs, particularly HDFS, to give a generally consistent
abstraction over multiple APIs.LVFS can allow you to shift existing or legacy Python applications from one storage API to another,
and has been used to move production applications from HDFS to GCS, and from local files to HDFS.LVFS doesn't require local temporary space; data doesn't need to save on disk when uploading or downloadingLVFS handles pretty complicated credential configurations, which can save a lot of time when developingLVFS integrates several big data formats used in Hadoop, so analytics applications usually can begin using
Hive or Spark data using only one line of code.LVFS uses Python modules whenever possible and only requiressshbe installed if you need to use SSH.
(This is because Paramiko does not support complex secured proxies.)Comparison against other solutionsLVFS is similar to some other storage tools:rclonercloneis a file sync utility that supports almost every imaginable storage API, with an interface very
similar torsync, except thatrsyncis older and only supports SSH and it's own custom protocol.Both systems create a common interface for multiple APIsLVFS separates reading from writing so you can read or write without touching local disk.
This is important for using LVFS inside a cluster because this reduces disk activity by at least half.
It is also important in containers, which often have very little available disk space (only memory)rclonesupports way more protocols, where LVFS only supports the most common ones.rcloneis a command line application, but LVFS is a Python library so integrating them into applications
is much different.hdfsThere is a Python package namedhdfswhich supports WebHDFS accesses from Python, and it is the most closely
aligned open source project to LVFS. It is essentially an SDK for WebHDFS and supports everythinglvfssupports, because LVFS useshdfs.LVFS supports more than one protocol, which is important if you are using LVFS to "lift and shift" a legacy
application between storage APIshdfssupports a more thorough and complete API around HDFS. In particular, it provides file-like objects
that can be useful if a file is larger than available memory or if you know you only need part of a file.If you are sure you don't need any other storage APIs, or you need more direct control at the expense
of some more difficulty porting to a new API,hdfsmay be a better choice.AlluxioAlluxio is a company and their eponymous suite of tools, which includes adapters between several kinds of
storage. Alluxio's scope is dramatically larger than LVFS and has many advantages and caveats:Alluxio can mount HDFS as a local volume on a Linux machine. This means every Unix command, includinglsandsqliteandawkand everything else, will be able to use HDFS or S3 or many other APIs.
This is areally amazing concept, and it works using the kernel's virtual file system rather than
an interface on a user-application level.There are a few open source projects trying to do this for HDFS as well, the completeness of which I
have not been able to verify. They deserve some attention as well.Posix file systems and these storage APIs are so much different in the details that actually relying
on these adapters in production is super risky, not because of anything wrong with Alluxio but on account
of specifics. What happens when one app has a file open and another deletes it, or howmmap()should
work after a processfork()'s? This is why bona-fide file systems take several years to develop.LVFS is not really an alternative to Alluxio. Alluxio is a massive project connecting so many APIs to
so many others, and requires a good bit of installation and know-how. LVFS requires a single configuration
file and a few lines of Python and effects only the applications that are built for it and import it.MinioMinio is a specific implementation of the Simple Storage Service protocol (S3). It provides very nice
S3 clients for several languages, including Python. It also supports an adapter server that exposes an
S3 endpoint for HDFS clusters. You could reasonably combine these to make a system where you can easily
migrate between HDFS and S3 (which many storage APIs support). This could be a good plan, but in our
case we needed to directly support GCS buckets with their SDK rather than also using S3 for
GCS, which we haven't investigated for feasibility.Installlvfsis not yet published to the PyPI repository. When it is, we'll update how to install it here.
Because PyPI already has a package namedlvfs, the package name will likely be different when it
does get published.ConfigureLVFS is designed firstly for HDFS, and sometimes HDFS clusters are firewalled and in a different subnet,
requiring an intermediary host to facilitate the communnication. There are also several ways you might
need to authenticate, and there are some cases where you even need to authenticate different ways
simultaneously. LVFS handles all the common paths encountered in a large corporate environment.LVFS configuration is stored in~/.config/lvfs.ymlbecause we expect that either you are in a container,
in which case the home directory is specific to an application, or you are using a laptop, where you will
likely want to handle development keys or credentials for several projects at once.Please note your company may already have documentation on how to configure LVFSConfiguration: Connect to an unauthenticated HDFS cluster behind an SSH tunnelThis is probably the most questionable setup but if you find yourself using a
Hadoop cluster that does not support authentication, but is instead protected
by keeping it on a separate subnet, you will need to connect to the subnet,
and usually that is done with SSH. LVFS supports this but it is a little involved.If you can, avoid doing this in production. SSH tunnels are not great for
either performance or reliability.# LVFS only needs credentials but we prefix with this in case we need additional information latercredentials:# In Yaml the - indiciates this is a list element.# Any additional stanzas will start with a similar --realm:# LVFS uses realms to determine which set of credentials to use for each URL.# Every credential stanza gets one realm.# LVFS will use the first stanza that matches the URL.# In this case we match anything using HDFSOverSSHclassname:HDFSOverSSH# Note how the indentation dropped by two spaces.# The rest of the stanza configures the connection.ssh_username:your_ssh_user# This is the username used to log into the jumpbox# LVFS will not pass SSH passwords. Use asymmetric keys instead.ssh_jump_host:dns_name# This is the actual jumpbox# Replace this with your Hadoop username used for HDFS;# this may be the same as your company's SSO.# Do not specify a password, even an empty one. This disables authentication.username:ltorvalds# Hadoop clusters usually use Zookeeper to maintain high availability.# So to handle multiple master nodes, just separate the URLs with spaces.# You may need to ask your admin for these URLs, or they may be available on a corporate wiki or FAQ.webhdfs_root:http://hadoopmaster001.corporate.com:50070;http://hadoopmaster002.corporate.com:50070;http://hadoopmaster003.corporate.com:50070Configuration: Connect to an unauthenticated HDFS cluster from within the subnetIf you use an unauthenticated HDFS cluster, then you should avoid poking holes in
your firewalls as much as possible, and you can help do that by keeping the data
processing within the cluster. In that case, configuration is a lot easier.credentials:-realm:# For historic reasons, all HDFS connections use the HDFSOverSSH connection class# because SSH will be disabled when you don't configure it here.classname:HDFSOverSSH# This is the Hadoop username; there is no jumpboxusername:ltorvalds# You may need to ask your admin for these URLs, or they may be available on a corporate wiki or FAQ.webhdfs_root:http://hadoopmaster001.corporate.com:50070;http://hadoopmaster002.corporate.com:50070;http://hadoopmaster003.corporate.com:50070Configuration: Connect using AD credentials to an HDFS cluster with open WebHDFS accessConnecting to HDFS using authenticated, encrypted WebHDFS is generally better practice
than encrypting using SSH. LVFS is compatible with this setup and you should find it to
be quite performant. The password is stored in the clear in this file. For that reason,be careful to control file permissionsand be sure only to use this for development
purposes. Production applications should probably use Kerberized service principals.credentials:-realm:# HDFS is really always HDFSOverSSH. It's a historical artifact.# In this case SSH is not used.classname:HDFSOverSSHusername:my_ad_usernamepassword:my_ad_password# Set this to the endpoint your cluster exposes.# Keep in mind, this is probably not a master node, and it probably doesn't use Zookeeper# Instead, this is probably a Knox server, the hint being it probably is at port 8443.webhdfs_root:https://theknoxserver.corporate.com:8443/gateway/defaultConfiguration: Connect using Kerberos service principals, inside an HDFS clusterThis is probably the most production worthy setup, where authentication is used,
as well as encryption, but actually nothing is leaving the subnet anyway.This does not set up Kerberos authentication, it just uses it.
You need to have successfully runkinitbefore running your app.credentials:-realm:# HDFS is really always HDFSOverSSH. It's a historical artifact.# In this case SSH is not used.classname:HDFSOverSSH# The username kerberos is special and will trigger Kerberos authentication# be sure to call `kinit` sometime before you run your app so that your# tickets will be set up already. LVFS will not run it for you.username:kerberoswebhdfs_root:https://themasterserver.corporate.com:9871NotesKerberos still requires that you runkinitsomehowbeforeyou start using LVFS,
and preferably before you even start Python. You need to figure out how to make that happen.
In an interactive terminal,kinitwill prompt you for your AD password,
and if you have a keytab it will not. For this reason, you will need a keytab for non-interactive
logins. Your case may be different, but most likely this keytab will be in a file somewhere,
and only accessible to certain Unix users due to file permissions. Ask your admins where and who.Finally, check withklistthat Kerberos is already working on this machine or container.
Afterkinithas been run, check thatklistshows the principals you expect;
If not there's no way LVFS will get it right either.It's not clear whether it is practical to authentiate with Kerberos with SSH tunneling,
because you would need to setup Kerberos there, which may require extensive configuration.
At any rate, this is not been tested with LVFS.For later reference, these are the possible modes for HDFSOverSSH:ssh_jump_hostssh_usernameusernamepasswordwebhdfs_rootuse case(any)(any)(any)(any)NoneInvalid, unconfigurednot NoneNone(any)(any)(any)InvalidNonenot None(any)(any)(any)Invalid(any)(any)Nonenot None(any)Invalidnot Nonenot Nonenot NoneNonenot NoneHDP2 with SSHNoneNonenot NoneNonenot NoneHDP2 without SSHnot Nonenot Nonenot Nonenot Nonenot NoneHDP3+AD with SSHNoneNonenot Nonenot Nonenot NoneHDP3+AD without SSHnot Nonenot NonekerberosNonenot NoneHDP3+Kerberos without SSH &NoneNonekerberosNonenot NoneHDP3+Kerberos with SSH(&) untestedExample codeRead a text file as YAML(URL.to(input("Where's the config file? ")).read_yaml())Read parquet format tables from HDFSYou can provide a file or a directory.
Directories (as in this case) will have all their shards concatenated into one dataframe.
Partitions are not concatenated, as they would usually be too large anyway.df=(URL.to("hdfs://hdfsmasternode.example.com/path/to/your/table").read_parquet())Copy files recursively from HDFS to localLVFS is designed for convenience, not speed or scalability, and each file will be buffered in memory.
Don't use this method if any of your files are more than a couple gigabytes each.
You'll run out of memory if you do.(URL.to("hdfs://hdfsmasternode.example.com/path/to/your/table").cp("/maybe/somewhere/local"))Create a configurable model on the spotJust to show how you might want to interact with URLs, here's an example where we load some
configuration files from any location (local or remote!) and then try to load a model from there.Keep in mind a lot of this is not about URL, it's just giving the example some context.defload_model(home:URL):""" Configure and read a model stored in a folder """composed_conf={}forconfinsorted(home.ls(recursive=True)):# Each configuration alphabetically. Use prefixes like "05-" to compose them.ifconf.basename().endswith(".yml"):# URLs are always absolute, so you can use ls() results immediately.composed_conf.update(conf.read_yaml())assert"latest_model"incomposed_confmodel=URL.to(composed_conf).read_pickle()returnmodel.compose_enhanced_sheaf_cohomologies(**composed_conf)if__name__=="__main__":importosfromargparseimportArgumentParserasAparser=A(description="Launch sheaf model")parser.add_argument("model",help="Whence come the models",default=os.getpwd())parser.add_argument("version",help="Model version",default="latest")parser.add_argument("data",help="Where the parquets are stored",default=os.getpwd())args=parser.parse_args()# You can tack on URL segments using `.join(..)`.# The right operand must be a string, not a URL, because URLs are absolute.model=load_model(URL.to(args.model).join(args.version))model.process(URL.to(args.data).read_parquet())print("All done!")ArchitectureLVFS is based on an abstract URL class with one important static method (to(..)).URL.to(..)checks the protocol and the machine hostname to see which connection implementation
is most appropriate for the location you're running and where you want to connect to.
It will give you an instance of that implementation seamlessly.There are a few methods that every implementation needs to implement, likestat(),ls(),
andread_binary()among a few others. But many methods, likeread_json()andcp()are defined
in terms of the other methods in order to reduce duplication. That said, GCS is strange and there
are many sharp edges you hopefully will not be cut by. Consider for example that you must always
buffer the fill content of a GCS blob in memory, which is why LVFS does not support streaming.API DocumentationYou can generate API documentation for this package after cloning it usingpdoc3through thegenerate-docs.shscript in the project root directory.