Saravanan666/pypi-org huggingface dataset:
Package: h5darkframes
H5DARKFRAMES (beta)H5Darkframes is a python library for generating and using darkframes library.
For now, it supports only asi zwo cameras (seehttps://github.com/MPI-IS/camera_zwo_asi).This is beta, and need some more testingInstallationfrom source:gitclonehttps://github.com/MPI-IS/h5darkframes.gitcdh5darkframes
pipinstall.from pypi:pipinstallh5darkframesUsageAssuming thatcamera-zwo-asiis installed and a camera is plugged:creating a darkframe libraryFirst, a configuration file must be created. In a terminal:darkframes-zwoasi-configThis will create in the current directory a filedarkframes.tomlwith a content similar to:[darkframes]
average_over = 5

[camera.controllables]
AutoExpMaxExpMS = 30000
AutoExpMaxGain = 285
AutoExpTargetBrightness = 100
BandWidth = "auto"
CoolerOn = 0
Exposure = 300
Flip = 0
Gain = 400
HighSpeedMode = 0
MonoBin = 0
Offset = 8
TargetTemp = 26
WB_B = 95
WB_R = 52

[camera.roi]
start_x = 0
start_y = 0
width = 4144
height = 2822
bins = 1
type = "raw8"

[darkframes.controllables.TargetTemp]
min = -15
max = 15
step = 3
threshold = 1
timeout = 600

[darkframes.controllables.Exposure]
min = 1000000
max = 30000000
step = 5000000
threshold = 1
timeout = 0.1

[darkframes.controllables.Gain]
min = 200
max = 400
step = 100
threshold = 1
timeout = 0.1You may edit this file to setup:your desired camera configurationthe controllables over which darkframes will be created, and over which rangethe number of pictures that will be averaged per darkframesFor example:[darkframes.controllables.TargetTemp]
min = -15
max = 15
step = 3
threshold = 1
timeout = 600implies that darkframes will be taken for values of temperature -15, -12, -9, ... up to +15.creating the darkframes libraryCall in a terminal:# change "mylibraryname" to a name of your choicedarkframes-zwoasi-library--namemylibrarynameYou may get stats regarding the library:darkframes-infousing the libraryimporth5darkframesasdarkimportcamera_zwo_asiaszwofrompathlibimportPath# path to the librarypath=Path("/path/to/darkframes.hdf5")# handle over the librarylibrary=dark.ImageLibrary(path)# handle over the cameracamera=zwo.Camera(0)# taking a picture. Image is an instance of zwo.Imageimage=camera.capture()# getting the current camera configurationcontrols=camera.get_controls()# "Temperature", "Exposure" and "Gain" must be the# controllables that have been iterated over# during the creation of the librarydarkframe_target={"Temperature":controls["Temperature"].value,"Exposure":controls["Exposure"].value,"Gain":controls["Gain"].value}# getting the darkframe that is the closest to the target# darkframe: a numpy array# darkframe_config: the config of the camera when the darkframe was takendarkframe,darkframe_config=library.get(darkframe_target)# optional sanity checksassertimage.get_data().shape==darkframe.shapeassertimage.get_data().dtype=darkframe.dtype# substracting the darkframesubstracted_image=image.get_data()-darkframe
Package: h5dataframe
h5dataframeDrop-in replacement for pandas DataFrames that allows to store data on an hdf5 file and manipulate data directly from that h5df file without loading it in memory.Warning !This is very much awork in progress, some features might not work yet or cause bugs.Saveyour data elsewhere before converting it to an H5DataFrame.If you miss a feature from pandas DataFrames, please fill an issue or feel free to contribute.OverviewThis library provides theH5DataFrameobject, replacing the regularpandas.DataFrame.AnH5DataFramecan be created from apandas.DataFrameor from a dictionnary of (column_name -> column_values).>>>importpandasaspd>>>fromh5dataframeimportH5DataFrame>>>df=pd.DataFrame({'a':[1,2,3],'b':[4,5,6]},index=['r1','r2','r3'])>>>hdf=H5DataFrame(df)>>>hdfabr114r225r336[RAM][3rowsx2columns]At this point, all the data is still loaded in RAM, as indicated by the second-to-last line. To write the data to an h5df file, use theH5DataFrame.write()method.>>>hdf.write('path/to/file.h5')>>>hdfabr114r225r336[FILE][3rowsx2columns]TheH5DataFrameis now backed on an h5df file, only loading data in RAM when requested.Alternatively, anH5DataFramecan be read directly from an previously created h5df file with theH5DataFrame.read()method.>>>fromh5dataframeimportH5Mode>>>H5DataFrame.read('path/to/file.h5',mode=H5Mode.READ)abr114r225r336[FILE][3rowsx2columns]The default mode isREAD('r') which creates aread-onlyH5DataFrame. To modify the data, usemode=H5Mode.READ_WRITE('r+').InstallationFrom pip:pipinstallh5dataframeFrom source:gitclonegit@github.com:Vidium/h5dataframe.git
Package: h5db
安装pip install -U h5db目前测试版本为python3，python2不支持入门读写数据例如需要储存和读取下面一个词典importh5dbimportnumpyasnp# make sure save_dir existsave_dir='./data'save_dat=[['1',np.random.random([10,1024])],['2',np.random.random([4,1024])]]h5db.simple_add(save_dir,save_dat)print(h5db.simple_search(save_dir,['1','2']))说明h5db默认使用简单hash函数将key分成100个部分，存储到100个对应的h5文件，搜索的时候会同时开启100个进程进行搜索，最后返回一个dict类型: {key: value}, 如果没有搜索到，则value是None；对于可以转换成int的类型的key, 默认的hash函数为 int(key) % 100, 对于str类型的key，默认用其sha1值的10位数 % 100；目前主要支持{str: np.ndarray} 和 {str: str}这两种数据类型，其他复杂类型未测试详细用法主要的类是H5classH5:'''
    H5DB的类
    save_dir: h5文件储存目录
    logger_path: logger储存目录
    l1_size: 一级分类个数
    l2_size: 二级分类个数（h5文件中的groups)
    hash_l1: 可以自定义一级hash函数
    hash_l2：可以自定义二级hash函数
    '''def__init__(self,save_dir:str=None,logger_path:str=None,l1_size:int=100,l2_size:int=2000,hash_l1:callable=None,hash_l2:callable=None):可以自定义key的hash函数，也可以使用默认的hash函数，默认的hash函数可以见gen_mod_hash和gen_mod_hash_shift主要作用的两个函数插入数据defmultiple_file_add(self,data_pairs:list,save_prefix:str='part.',replacement:bool=False):data_pairs：[k, v]对的list
replacement：遇到重复的key，value对，是否进行覆盖搜索数据defsearch(self,vids:list,save_prefix:str='part.',max_parallel:int=None):save_prefix必须与插入数据时候设定的prefix一致max_parallel最高的并发数量，注意到本身是多文件存储的，因此最低并发数是初始化类时的l1_size，注意当io是瓶颈时，继续增大max_parallel并不会加快搜索速度默认使用multiprocessing.Manager.dict进行多进程沟通，也可以自己修改代码使用其他框架。一个基础例子fromh5dbimportH5importjsonh5=H5(save_dir='./data',l1_size=10)file_to_save={'1':{'name':'lr','age':27},'2':{'attr':'cd','attr-2':26.5},}# 注意到这里必须转换成str类型，dict类型不支持file_to_save=[[k,json.dumps(v)]fork,vinfile_to_save.items()]h5.multiple_file_add(file_to_save)print(h5.search(['1','2','3']))输出11/26/2020 19:40:01-INFO-Filename ./data/part.1.h5 will write 1 lines
11/26/2020 19:40:01-INFO-Filename ./data/part.2.h5 will write 1 lines
11/26/2020 19:40:01-INFO-dumping vids to idx.bin
11/26/2020 19:40:01-INFO-all Processes finished!
[('1', b'{"name": "lr", "age": 27}'), ('2', b'{"attr": "cd", "attr-2": 26.5}')]
Package: h5dict
H5dictAn interface to read/write HDF5 files as if they where dictionaries.** WARNING: ** Version 0.2.0 of h5dict isNOTfully compatible with previous versions.Instalationh5dict can be installed with pip:python -m pip install h5dictManual instalationClone the repository and run setup.py:git clone https://gitlab.com/nanogennari/h5dict.git
cd h5dict
python setup.py installUsageUsage example:import h5dict
import numpy as np

hdf5 = h5dict.File("file.hdf5", "r+")
hdf5["test_1"] = np.arange(100)
hdf5["test_2"] = {
    "names": ["Alice", "Bob"],
    "ages": [30, 25]
}

for keys in hdf5["test_2"]["names"].keys()
    print(hdf5["test_2"]["names"][key])

hdf5.close()DocumentationDocumentation can be foundhere.
Package: h5features
h5featuresThe h5featurespython packageprovides easy to use and efficient
storage oflarge features dataon the HDF5 binary file format.Installation from condaconda install-ccoml h5featuresInstallation from pippip install h5featuresInstallation from sourcesdependenciesThe package depends onnumpy,scipyandh5py. They can be
automatically installed by the setup script but it takes a long time
(full compilation). You may want to install the dependancies through
your system package manager.On Debian/Ubuntu:sudo apt-get install python3-numpy python3-scipy python3-h5pyUsing Python anaconda:conda install numpy scipy h5pyh5featuresInstall it from the sources with:python setup.py build && python setup.py installDocumentationSee the complete documentationonlineOr build it with:pip install Sphinx
cd docs && make htmlThe home page of the compiled documentation isdocs/_build/html/index.html.TestThe package comes with a unit-tests suit. To run it, first installpyteston your Python environment:pip install pytestThen run the tests with:pytest -v ./test
Package: h5glance
H5Glancelets you explore HDF5 files in the terminal or an HTML interface.Install it with:pip install h5glanceIn the terminal, you can get a tree view of a file:$ h5glance sample.h5
sample.h5
└path
  └inside
    └file   [float64: 100 × 100]The names of datasets, groups and links are colour-coded by default.
If you want to disable this, set the environment variableH5GLANCE_COLORS=0.Inspect a group or dataset inside it:$ h5glance sample.h5 path/inside/file
sample.h5/path/inside/file
      dtype: float64
      shape: 100 × 100
   maxshape: 100 × 100
     layout: Contiguous

sample data:
[[-0.27756437  0.36923643 -0.28113527 ...In bash & zsh, h5glance offers tab-completion for the paths inside HDF5 files.
To set this up, run:python -m h5glance.completerAlternatively, use-as the second argument, and h5glance will prompt you
for the object path with tab completion:$ h5glance sample.h5 -
Object path: sample.h5/  # try tapping tabHTML interfaceThe HTML interface lets you inspect HDF5 files in a Jupyter Notebook.Demo.ipynbshows how to use it.Why H5Glance?There are plenty of other tools to view HDF5 files, includingHDFViewandViTables, as well as various web-based viewers in
development. Why might you choose H5Glance?Practical terminal interface: if you’re working in a terminal, it’s much
quicker to use something there than to start a GUI application and click
through it.Tab completions are part of this - take a moment to set them up (see above).Static Jupyter views: H5Glance shows your HDF5 objects in simple HTML, which
doesn’t talk to the server. Export your notebook to HTML, orview it on
nbviewer,
and the H5Glance view is still there.Deeply nested structure: It was written atEuropean XFEL,
where data files can easily have 6 layers of nested groups. It tries to make
working with that easy.Some things it’s not designed for:Viewing data: The terminal interface can show raw data, but it focuses on the
structure of HDF5 files, not their content. H5Glance won’t show you plots or
images.Machine-readable output: It’s meant for people, not programs.
In Python, your code can useh5py.
For shell pipelines, use tools likeh5lsandh5dump.
Package: h5grove
h5grove, core utilities to serve HDF5 file contentsh5groveis a Python package that provides utilities to design backends serving HDF5 file content: attributes, metadata and data. HDF5 files are accessed withh5py.DocumentationThe documentation of the latest release is availablehere.RationaleThere are several packages out there that can serve HDF5 files. However, they are dedicated to their usecases and settle on one implementation, hampering reusability.In addition, some problems arise constantly when designing HDF5 backends. To name a few:Resolving external linksDealing with compression and slicing of datasetsEncoding data efficiently and consistently (looking at you,NaN,Infinityin JSON)h5groveaims at providing building blocks that solve these common problems and can be reused in existing or new backends.Installationpipinstallh5groveYou can useh5grovelow-level utilities whatever the backend implementation you choose. Additional utilities are provided for several Python web frameworks but require additional packages that can be installed the following way:FastAPIpipinstallh5grove[fastapi]Flaskpipinstallh5grove[flask]Tornadopipinstallh5grove[tornado]
Package: h5image
h5imageThis package provides standard functions for interacting with HDF5 files as
part of the CriticalMAAS project.This package will create HDF5 files with the following structure:group : all data associated with a single mapjson: the raw json file used to create the map and layersmap: the actual map imagedata: the actual bytes of the imagecorners: the corners of the map where layers have datapatches: for each layer (key) a list of patches as a tuple (x, y)layers_patch: for each patch (key is x_y) a list of layersvalid_patche: a list of locations across all layers as a tuple (x, y)<layer>: a layer of the map, the name of the layer is the name of filedata: the actual layer datapatches: a list of patches as a tuple (x, y) for this specific layerInstallationInstall using pip:pip install h5imageTo convert a folder of images to a HDF5 file use theh5createprogram.Quickstart exampleThe following code will create a new HDF5 file with a map and a layer, find the
patch with most layers, and load the map and the layers at that patch.importmatplotlib.pyplotaspltimportnumpyasnpimportrasteriofromh5imageimportH5Image# create map filemap="CA_Sage"h5i=H5Image(f"{map}.hdf5","w")h5i.add_image(f"{map}.json",folder="./data")# get biggest patchpatches=h5i.get_patches(map,True)sorted_patches={k:vfork,vinsorted(patches.items(),key=lambdaitem:len(item[1]),reverse=True)}loc,loc_layers=next(iter(sorted_patches.items()))row=int(loc.split("_")[0])col=int(loc.split("_")[1])print(f"Biggest number of layers ({len(loc_layers)}) for{map}is at ({row},{col})")# plot maprgb=h5i.get_patch(row,col,map)plt.imshow(rgb)plt.title('map')plt.axis('off')plt.show()# plot layers and legendforlayerinloc_layers:rgb=h5i.get_patch(row,col,map,layer=layer)plt.imshow(rgb,cmap=plt.colormaps['plasma'])plt.title(layer)plt.axis('off')plt.show()rgb=h5i.get_legend(map,layer)plt.imshow(rgb)plt.title("legend")plt.axis('off')plt.show()# saving image as geotiff# can also use h5i.save_image(map, 'map')dset=h5i.get_map(map)ifdset.ndim==3:image=dset[...].transpose(2,0,1)# rasterio expects bands firstelse:image=np.array(dset[...],ndmin=3)rasterio.open(f"{map}.tif",'w',driver='GTiff',compress='lzw',height=image.shape[1],width=image.shape[2],count=image.shape[0],dtype=image.dtype,crs=h5i.get_crs(map,layer),transform=h5i.get_transform(map,layer)).write(image)# close fileh5i.close()
Package: h5io
h5ioh5io is a package designed to facilitate saving some standard Python
objects into the forward-compatible HDF5 format. It is a higher-level
package thanh5py.Get the latest codeTo get the latest code using git, simply type:git clone git://github.com/h5io/h5io.gitIf you don’t have git installed, you can download a zip or tarball
of the latest code:https://github.com/h5io/h5io/archives/mainInstall h5ioAs any Python packages, to install h5io, go in the source code directory
and do:python setup.py installor if you don’t have admin access to your python setup (permission denied
when install) use:python setup.py install --userYou can also install the latest release version with pip:pip install h5io --upgradeor for the latest development version (the most up to date):pip install -e git+https://github.com/h5io/h5io#egg=h5io-dev --userDependenciesThe required dependencies to build the software areh5pyandnumpy.scipyis required for sparse matrix IO support.Licensingh5io isBSD-licenced(3 clause):This software is OSI Certified Open Source Software.
OSI Certified is a certification mark of the Open Source Initiative.Copyright (c) 2011, authors of h5io
All rights reserved.Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.Neither the names of h5io authors nor the names of any
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.This software is provided by the copyright holders and contributors
“as is” and any express or implied warranties, including, but not
limited to, the implied warranties of merchantability and fitness for
a particular purpose are disclaimed. In no event shall the copyright
owner or contributors be liable for any direct, indirect, incidental,
special, exemplary, or consequential damages (including, but not
limited to, procurement of substitute goods or services; loss of use,
data, or profits; or business interruption) however caused and on any
theory of liability, whether in contract, strict liability, or tort
(including negligence or otherwise) arising in any way out of the use
of this software, even if advised of the possibility of such
damage.
Package: h5json
hdf5-json=========.. image:: https://travis-ci.org/HDFGroup/hdf5-json.svg?branch=develop:target: https://travis-ci.org/HDFGroup/hdf5-jsonSpecification and tools for representing HDF5 in JSONIntroduction------------This repository contains a specification, library, and utilities for describing HDF5 content in JSON.The utilities can be used to convert any HDF5 file to JSON or from a JSON file (using the conventiondescribed here to HDF5).The library is useful for any Python application that needs to translate between HDF5 objects and JSONserializations.  In addition to the utilities provided in this repository, the library is used by HDFServer (a RESTful web service for HDF5), and HDF Product Designer (an application for creating productdesigns).This respository also include utilities to generate code in Fortran or Python based on a JSON file.See :doc:`Utilities` for a description of the included utilties.Websites--------* Main website: http://www.hdfgroup.org* Source code: https://github.com/HDFGroup/hdf5-json* Mailing list: hdf-forum@lists.hdfgroup.org <hdf-forum@lists.hdfgroup.org>* Documentation: http://hdf5-json.readthedocs.orgRelated Projects----------------* HDF Server: https://www.hdfgroup.org/projects/hdfserver/* Product Designer: https://wiki.earthdata.nasa.gov/display/HPD/HDF+Product+DesignerInstalling-----------Via pip::pip install h5jsonFrom a release tarball or Git checkout::python setup.py installpython testall.py  # optionalThe hd5tojson.py and jsontoh5.py convertors will be added to the path.See docs/Installation.rst for step by step instructions.Uninstalling------------Just remove the install directory and all contents to uninstall.Reporting bugs (and general feedback)-------------------------------------Create new issues at http://github.com/HDFGroup/hdf5-json/issues for any problems you find.For general questions/feedback, please use the list (hdf-forum@lists.hdfgroup.org).
Package: h5logger
Minimalist h5 Python loggerh5logger is a minimalist logger that allows to continuously save arbitrary quantities into h5 files. The logger automatically create/open/close h5 files and append data as a stream whenever the user calls thelogmethod of theloggerclass. Only dependancy requirement ish5py.Walkthrough exampleHere is a simple example to store different
realisation of random variables:import numpy as np
from h5logger import h5logger


# set seed for reproducibility
np.random.seed(0)

# create the logger that will save the continuous
# stream of data into the logging_data.h5 file
logger = h5logger("logging_data.h5")

for i in range(10):
    number = np.random.randint()

    # save the number realisation into the `number`
    # variable, any string can be used for name
    logger.log('number', number)

# all the data is now into the h5 file which can be
# access as desired by the user
# h5logger has some built-in reading functions
with logger.open() as data:
    print(data['accu'])
    #Minimalist Python logger/saver storing continuous stream of data into h5 filesWhy the h5 format?h5 files have many advantages:big data friendely:h5py does not return an in-memory numpy array. Instead it returns something that behaves like it, hence accessing different chunks of data from any saved quantity can be done near instantly even though the entire dataset might be humongously large. Have a look at theh5py introductionfor more information.suited for saving streaming data:because h5 supports saving data per chunk, it is extremely well suited to dynamically expand dataset and insert new observations, i.e., it efficiently deals with saving streams of datacommand-line monitoring:another great advantage of h5 files is their ability to be monitored from a terminal with simple command-line e.g.h5ls logging_data.h5. This is a crucial feature to easily monitor (locally and remotely) the logging status. For a list of commands seethis guide(requires hdf5 package which can be installed withanaconda/brew/apt)
Package: h5massembly
step_to_h5mPlease note that this tool will likely change its name soon.This is intended to be a python package heavily inspired byParamak, and borrowing a lot from its namesake:https://github.com/fusion-energy/step_to_h5m).
It's "raison d'etre" is to enable a link between the online CAD-tool onShape and the neutron transport code OpenMC.We will use cadQuery and its links to OCCT to enable import and imprinting/merging algorithms.The code structure we intend is a main class Assembly which contains the geometry in terms of a set of volumes.
We then use gmsh to generate a surface mesh which may be used by OpenMC through DAGMC/MOAB (i.e. .h5m-files)To install/set up in a virtual python environmentreplace <name> with an arbitrary name for your virtual environmentClone the github repository as you'd normally doIn the directory where you want your environment to reside do:python -m venv <name>Activate the environment:source <name>/bin/activateinstall the requirements:pip install -r requirements.txtBuild and install moab (if not already installed). The moab team relies on conda for standard installation but are working on a pip-based solution. Once that is done moab would simply be added to the requirements-file instead.Clone the moab code-repository: e.g.git clone git@bitbucket.org:fathomteam/moab.gitConfigure and build the code:mkdir build; cd build; cmake .. -DENABLE_PYMOAB=1; make; make installRun a test case:The follwing code-snippet can now be run to explore the capabilities of Assembly/step_to_h5mimport assembly as ab
  a=ab.Assembly()
  a.stp_files=["file.step"]
  a.import_stp_files()
  a.export_brep('file.brep')
  a.brep_to_h5m(brep_filename='file.brep',min_mesh_size=0.1, max_mesh_size=10, samples=20)N.b. the last 3 parameters to brep_to_h5m are simply echoing their default values, and uses the default mesher: gmsh. Their significance is that all curves are sampled by 20 points, but limits the generated mesh-elements to be  within the size range [min_mesh_size,max_mesh_size].
This procedure will in turn call OCP and gmsh to produce a mesh with merged surfaces in the output file "dagmc.h5m"The other available meshing backend is the stl-export from CadQuery (accessible by settingbackend='stl') which uses the parametersstl_tolandstl_ang_tolto set meshing quality.Theexport_brep-step may be omitted, in which case a temporary file will be written.
Package: h5max
h5maxA utility package built uponh5pyfor easier data saving and loading of sparse data objects.h5maxhandles storing and loading ofscipy.sparsedata structures inh5pyfile objects, which is not natively supported. It assumes a simple data structure where information of individual samples are stored according to the index they occupy within datasets.🔗 Installationpipinstallh5max📖 User guideimporth5pyimporth5maximportnumpyasnpfh=h5py.File('my_data.h5','w')a=np.zeros((100,100))b=np.zeros((1000,50))a[7,1]=1b[1,0]=10m_list=[a,b]# store both a, bh5max.store_sparse(fh,m_list,format='csr')# load only a (index 0)a_out=h5max.load_sparse(fh,0,format='csr')# load [a,b]m_list_out=h5max.load_sparse(fh,[0,1],format='csr',to_numpy=True)# load all idxs in the datam_list_out=h5max.load_sparse(fh,format='csr')fh.close()✔️ Package featuresSupport forcsr,csc,coosparse typesSupport forbsr,dia,dok,lilsparse typesSupport for overwritingFlexible data loading and saving (both as sparse and numpy arrays.)Automatic format detection