Package: avroknife
AboutThis project provides a command-line utility for browsing and simple
manipulation of Avro-based files.Apache Avrois a serialization format
invented to be a language-independent way of communication betweenHadoopdata processing tasks. Hadoop
tasks produce output that, on an abstract level, can be regarded as a
list of objects of the same type. In practice, when using Avro format,
this list is represented in the file system as a directory containing
many Avro-formatted files where each file has the same schema. We call
this directory theAvro data store.avroknifeallows for browsing and simple manipulation of Avro data
stores. It was inspired by Avro library’s own tool calledavro-toolswhich is distributed with that library as a*.jarfile. Apart from
differences in particular functionalities, the main philosophical
difference between these two is thatavroknifeoperates on whole
Avro data store whileavro-toolsoperates on individual Avro files.FeaturesAccesses Avro data stores placed in the local file system as well as
in the Hadoop Distributed File System (HDFS).Note that in order to access HDFS, you need to havepydoopPython package installed.Provides the following execution modes (runavroknife-hfor
details):prints out schema of data store,dumps data store as JSON,dumps selected records from data store as a new data store,dumps a field from selected records to file system or to stdout,prints number of records inside a data store.Allows for simple selection of the records to be accessed based on
combination of the following constraints:index range of the records,limit set on number of returned records,value of a field.Usage examplesLet’s assume that we have an Avro data store available as/user/$USER/example_data_storedirectory in HDFS:$ hadoop fs -ls example_data_store
Found 4 items
-rw-r--r--   1 mafju supergroup        408 2014-09-18 11:36 /user/mafju/example_data_store/part-m-00000.avro
-rw-r--r--   1 mafju supergroup        449 2014-09-18 11:36 /user/mafju/example_data_store/part-m-00001.avro
-rw-r--r--   1 mafju supergroup        364 2014-09-18 11:36 /user/mafju/example_data_store/part-m-00002.avro
-rw-r--r--   1 mafju supergroup        429 2014-09-18 11:36 /user/mafju/example_data_store/part-m-00003.avroFirst, let’s check the schema of the data store$ avroknife getschema example_data_store
{
    "namespace": "avroknife.test.data",
    "type": "record",
    "name": "User",
    "fields": [
        {
            "type": "int",
            "name": "position"
        },
        {
            "type": "string",
            "name": "name"
        },
        {
            "type": [
                "int",
                "null"
            ],
            "name": "favorite_number"
        },
        {
            "type": [
                "string",
                "null"
            ],
            "name": "favorite_color"
        },
        {
            "type": [
                "bytes",
                "null"
            ],
            "name": "secret"
        }
    ]
}Then, let’s list all its records:$ avroknife tojson example_data_store
{"position": 0, "name": "Alyssa", "favorite_number": 256, "favorite_color": null, "secret": null}
{"position": 1, "name": "Ben", "favorite_number": 4, "favorite_color": "red", "secret": null}
{"position": 2, "name": "Alyssa2", "favorite_number": 512, "favorite_color": null, "secret": null}
{"position": 3, "name": "Ben2", "favorite_number": 8, "favorite_color": "blue", "secret": "MDk4NzY1NDMyMQ=="}
{"position": 4, "name": "Ben3", "favorite_number": 2, "favorite_color": "green", "secret": "MTIzNDVhYmNk"}
{"position": 5, "name": "Alyssa3", "favorite_number": 16, "favorite_color": null, "secret": null}
{"position": 6, "name": "Mallet", "favorite_number": null, "favorite_color": "blue", "secret": "YXNkZmdm"}
{"position": 7, "name": "Mikel", "favorite_number": null, "favorite_color": "", "secret": null}Now, let’s select the records where thefavorite_colorattribute is
equalblueand the index of the record is 5 of larger:$ avroknife tojson --select favorite_color="blue" --index 5- example_data_store
{"position": 6, "name": "Mallet", "favorite_number": null, "favorite_color": "blue", "secret": "YXNkZmdm"}Next, let’s extract value of thenameattribute for all records
where thefavorite_colorattribute is equalblue:$ avroknife extract --value_field name --select favorite_color="blue" example_data_store
Ben2
MalletNote that if the data store was placed in the local file system, you
would have to prefix its path withlocal:, e.g.$ avroknife tojson local:example_data_storeThat’s it. Runavroknife-hto find out more about other modes and
options ofavroknife.InstallationThe project is available in the PyPI repository, so in oder to install
it, you need to dosudo pip install avroknifeIf you want to access HDFS,pydoopPython library needs to be
installed in the system. You can follow the description onPydoop’s
documentation pagein order to proceed with its installation. On Ubuntu 14.04, installing
Pydoop boils down to the following steps:Install Hadoop. If you want to install it on a single node in a
so-called pseudo-distributed mode, I recommend to use the Cloudera
Hadoop distribution. This can be done by following Cloudera’sstep-by-step
guide.
Apart from thehadoop-0.20-conf-pseudopackage from the Cloudera
repository that is mentioned in the guide, you also have to installhadoop-clientpackage.Make sure that Java JDK is installed correctly. This can be done by
executing the following steps.Make sure that Java JDK is installed. This can be done by
installingopenjdk-7-jdkpackage, i.e.,sudoapt-getinstallopenjdk-7-jdk.Make sure that theJAVA_HOMEenvironment variable is set
properly. This can be done by adding lineexportJAVA_HOME="/usr/lib/jvm/default-java"in/etc/profile.d/my_env_vars.shfile.Install the following Ubuntu packages:python-dev,libssl-dev, i.e.,sudoapt-getinstallpython-devlibssl-dev.Install the Pydoop package throughpip, i.e.,sudo-ipip install pydoop.TroubleshootingOn my system (Ubuntu 14.04) with my installation of Hadoop (CDH 4.7.0),
the following message was printed on stderr every time that I accessed
HDFS:SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further detailsIt turned out that among the jars loaded by thepydooplibrary, theslf4jjar was missing (the symbolic link to it was broken). In order
to amend this problem Iremoved the broken symbolic link withsudo rm/usr/lib/hadoop/client/slf4j-log4j12.jarcreated a correct symbolic link withsudo ln-s/usr/share/java/slf4j-log4j12.jar/usr/lib/hadoop/client/slf4j-log4j12.jar(you need to have thelibslf4j-javapackage installed in order to
have the target jar file present).HistoryThe initial version ofavroknifewas created in March 2013. The
script has been used by the developers of the Information Inference
Service in theOpenAIREplusproject.LicenseThe code is licensed under Apache License, Version 2.0