Package: v3io
Python SDK for V3IOPython (2.7 and 3.5+) client for the Iguazio Data Science Platform (the "platform"). Designed to allow fast access to the data layer and basic access to the control layer. This library can be used in Nuclio functions, Jupyter notebooks, local Python IDEs; anywhere with a Python interpreter and access to a platform.InstallingSimply get withpip(locking to a specific version is recommended, as always):pip install v3ioDataplane clientWith the dataplane client you can manipulate data in the platform's multi-model data layer, including:ObjectsKey-values (NoSQL)StreamsContainersUnder the hood, the client connects through the platform's web API (https://www.iguazio.com/docs/latest-release/data-layer/reference/web-apis/) and wraps each low level API with an interface. Calls are blocking, but you can use the batching interface to send multiple requests in parallel for greater performance.Creating a clientCreate a dataplane client, passing in the web API endpoint and your access key:importv3io.dataplanev3io_client=v3io.dataplane.Client(endpoint='https://v3io-webapi:8081',access_key='some_access_key')Note: In some environments (like Jupyter notebooks) you do not need to pass the endpoint and access key, as they will automatically be inferred from the environmentMaking requestsThe client supports a handful of low level APIs, each receiving different arguments but all returning aResponseobject. AResponseobject holds 4 fields:status_code: The HTTP status code returnedoutput: An object containing the parsed response (each API returns a different object)headersandbody: The raw headers and body of the responseYou would normally only access theoutputfield unless an API was called that returns raw data likeobject.get(in which casebodyholds the response). Consult the reference for each API call to see how to handle itsResponseobject. In the example below, we perform a simple request to get the containers available in our tenant, print the returned status code and containers:# list the contents of a containerresponse=v3io_client.container.list('users','/')# print the status code. outputs:##  Status code: 200#print(f'Status code:{response.status_code}')We can also get help information about the parameters this API call receives:help(v3io_client.container.list)Handling errorsBy default, making a request will raise an exception if any non-200 status code is returned. We can override this default behavior in two ways.The first is to simply never raise an exception and handle the status manually:# list the contents of a container and never raise an exceptionresponse=v3io_client.container.list('users','/',raise_for_status=v3io.dataplane.RaiseForStatus.never)# do anything we want with the status code# some_logic(response.status_code)The second is to indicate which status codes are acceptable:# list the contents of a container and raise if the status code is not 200 or 204response=v3io_client.container.list('users','/',raise_for_status=[200,204])Creating batchesTo get the highest possible throughput, we can send many requests towards the data layer and wait for all the responses to arrive (rather than send each request and wait for the response). The SDK supports this through batching. Any API call can be made through the client's built inbatchobject. The API call receives the exact same arguments it would normally receive (except forraise_for_status), and does not block until the response arrives. To wait for all pending responses, callwait()on thebatchobject:# do 16 writes in parallelforidxinrange(16):# returns immediatelyv3io_client.batch.object.put(container='bigdata',path=f'/object{idx}',body=f'object-{idx}')# wait for all writes to completev3io_client.batch.wait()The loopedobject.putinterface above will send 16put objectrequests to the data layer in parallel. Whenwaitis called, it will block until either all responses arrive (in which case it will return aResponsesobject, containing theresponsesof each call) or an error occurs - in which case an exception is thrown. You can passraise_for_statustowait, and it behaves as explained above.Note: Thebatchobject is stateful, so you can only create one batch at a time. However, you can create multiple parallel batches yourself through the client'screate_batch()interfaceExamplesAccessing objectsPut data in an object, get it back and then delete the object:# put contents to some objectv3io_client.object.put(container='users',path='/my-object',body='hello, there')# get the objectresponse=v3io_client.object.get(container='users',path='/my-object')# print the contents. outputs:##   hello, there#print(response.body.decode('utf-8'))# delete the objectv3io_client.object.delete(container='users',path='/my-object')For a detailed example, refer to thev3io-objects notebook.Accessing key-values (NoSQL)Create a table, update a record and run a query.items={'bob':{'age':42,'feature':'mustache'},'linda':{'age':41,'feature':'singing'},'louise':{'age':9,'feature':'bunny ears'},}# add the records to the tableforitem_key,item_attributesinitems.items():v3io_client.kv.put(container='users',table_path='/bobs-burgers',key=item_key,attributes=item_attributes)# adds two fields (height, quip) to the louise recordv3io_client.kv.update(container='users',table_path='/bobs-burgers',key='louise',attributes={'height':130,'quip':'i can smell fear on you'})# get a record by key, specifying specific argumentsresponse=v3io_client.kv.get(container='users',table_path='/bobs-burgers',key='louise',attribute_names=['__size','age','quip','height'])# print the item from the response. outputs:##   {'__size': 0.0, 'quip': 'i can smell fear on you', 'height': 130.0}#print(response.output.item)# create a query, and use an items cursor to iterate the resultsitems_cursor=v3io_client.kv.new_cursor(container='users',table_path='/bobs-burgers',attribute_names=['age','feature'],filter_expression='age > 15')# print the outputforiteminitems_cursor.all():print(item)For a detailed example, refer to theexamples/v3io-kv notebook.Accessing streamsCreates a stream with several partitions, writes records to it, reads the records and deletes the stream:# create a stream w/8 shardsv3io_client.stream.create(container='users',stream_path='/my-test-stream',shard_count=8)# write 4 records - 3 with explicitly specifying the shard and 1 using hashingrecords=[{'shard_id':1,'data':'first shard record #1'},{'shard_id':1,'data':'first shard record #2'},{'shard_id':2,'data':'second shard record #1'},{'data':'some shard record #1'}]v3io_client.stream.put_records(container='users',stream_path='/my-test-stream',records=records)# seek to the beginning of the shard of #1 so we know where to read fromresponse=v3io_client.stream.seek(container='users',stream_path='/my-test-stream',shard_id=1,seek_type='EARLIEST')# get records from the shard (should receive 2)response=v3io_client.stream.get_records(container='users',stream_path='/my-test-stream',shard_id=1,location=response.output.location)# print the records. outputs:##   first shard record #1#   first shard record #2#forrecordinresponse.output.records:print(record.data.decode('utf-8'))# delete the streamv3io_client.stream.delete(container='users',stream_path='/my-test-stream')For a detailed example, refer to thev3io-streams notebook.Support for asyncio (experimental)All synchronous APIs are available asasyncinterfaces through theaiomodule. The differences between the sync and async API is as follows:You must initialize a different client (v3io.aio.dataplane.Client) fromv3io.aio.dataplaneAll interfaces should be called withawaitv3io.aio.dataplane.RaiseForStatus.nevershould be used overv3io.dataplane.RaiseForStatus.never(although they are the same)The batching functionality doesn't exist, as you can achieve the same through standard asyncio practicesNote: For the time being,aiohttpmust be provided externally - it is not a v3io-py dependency. This will be fixed in future versions.importv3io.aio.dataplanev3io_client=v3io.aio.dataplane.Client(endpoint='https://v3io-webapi:8081',access_key='some_access_key')# put contents to some objectawaitv3io_client.object.put(container='users',path='/my-object',body='hello, there')# get the objectresponse=awaitv3io_client.object.get(container='users',path='/my-object')# print the contents. outputs:##   hello, there#print(response.body.decode('utf-8'))# delete the objectawaitv3io_client.object.delete(container='users',path='/my-object')Support for unit tests (experimental)To facilitate unit tests, v3io-py can pass along all requests to the user rather than communicate over the network with the dataplane layer. This is done by passing theClientobject averifiertransport. This transport receives a list of callables that will be called in the order of requests theClientgenerates. The user can then verify the content of the request and either return a response or raise an exception. With this mechanism, users can test snippets of their code for various positive and negative behvior.In the below example, two requests are sent and verified. UsingMagicMockfor outputs/responses is recommended:deftest_verifier_transport(self):container_name='some_container'## Register a set of verifiers#def_verify_object_get(request):# verify some stuff from the requestself.assertEqual(request.container,container_name)self.assertEqual(request.path,os.path.join(os.sep,container_name,'some/path'))# return a mocked responsereturnunittest.mock.MagicMock(status_code=200,body='some body')def_verify_kv_get(request):# verify some stuff from the requestself.assertEqual(request.container,container_name)self.assertEqual(request.path,os.path.join(os.sep,container_name,'some/table/path/some_item_key'))# prepare and output mockoutput=unittest.mock.MagicMock(item={'some_key':'some_value'})# prepare a response mockreturnunittest.mock.MagicMock(output=output)# create a verifier transport. pass it a set of request verifiersverifier_transport=v3io.dataplane.transport.verifier.Transport(request_verifiers=[_verify_object_get,_verify_kv_get,])## Run some flow. This can be a call to the actual code you want to test. By specfiying all# the verifying behavior beforehand, the actual code under test remains untouched## create a client with a verifier transportclient=v3io.dataplane.Client(transport_kind=verifier_transport)# do an object getresponse=client.object.get(container='some_container',path='some/path')# verify that we got some body from the verifierself.assertEqual(response.body,'some body')self.assertEqual(response.status_code,200)# do an item getresponse=client.kv.get(container=container_name,table_path='some/table/path',key='some_item_key')# verify that we got a proper responseself.assertEqual(response.output.item['some_key'],'some_value')Custom transportsIf theverifiertransport is inadequate for your use case, a custom transport can be provided. It must conform to the following minimal interface:classTransport(v3io.dataplane.transport.abstract.Transport):def__init__(self):super().__init__(None,'',0,None,'DEBUG')# receives a request and returns a responsedefwait_response(self,request,raise_for_status=None):pass