Saravanan666/pypi-org huggingface dataset:
Package: dolma
Dolma is two things:Dolma Dataset: an open dataset of 3 trillion tokens from a diverse mix of web content, academic publications, code, books, and encyclopedic materials.Dolma Toolkit: a high-performance toolkit for curating datasets for language modeling -- this repo contains the source code for the Dolma Toolkit.Dolma DatasetDolma is an open dataset of 3 trillion tokens from a diverse mix of web content, academic publications, code, books, and encyclopedic materials.
It was created as a training corpus forOLMo, a language model from theAllen Institute for AI(AI2).Dolma is available for download on the HuggingFace 🤗 Hub:huggingface.co/datasets/allenai/dolma. To access Dolma, users must agree to the terms ofAI2 ImpACT License for Medium Risk Artifactson theHuggingFace 🤗 Hub. Once agreed you can follow the instructionshereto download the dataset.You can also read more about Dolma inour announcement, as well as by consulting itsdata sheet.Dolma ToolkitThis repository houses the Dolma Toolkit, which enables curation of large datasets for (pre)-training ML models. Its key features are:High Performance⚡: Can process billions of documents concurrently thanks to built-in parallelism.Portability🧳: Works on a single machine, a cluster, or cloud environment.Built-In Taggers🏷: Includes ready-to-use taggers commonly used to curate datasets such asGopher,C4, andOpenWebText.Fast Deduplication🗑: Speedy document deduplication using a Rust Bloom filter.Extensibility🧩 &Cloud Support☁: Supports custom taggers and AWS S3-compatible locations.To install, simply typepip install dolmain your terminal.To learn more about how to use the Dolma Toolkit, please visit thedocumentation.CitationIf you use the Dolma dataset or toolkit, please cite the following items:@article{dolma,title={{Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research}},author={Luca Soldaini and Rodney Kinney and Akshita Bhagia and Dustin Schwenk and David Atkinson and Russell Authur and Ben Bogin and Khyathi Chandu and Jennifer Dumas and Yanai Elazar and Valentin Hofmann and Ananya Harsh Jha and Sachin Kumar and Li Lucy and Xinxi Lyu and Ian Magnusson and Jacob Morrison and Niklas Muennighoff and Aakanksha Naik and Crystal Nam and Matthew E. Peters and Abhilasha Ravichander and Kyle Richardson and Zejiang Shen and Emma Strubell and Nishant Subramani and Oyvind Tafjord and Evan Pete Walsh and Hannaneh Hajishirzi and Noah A. Smith and Luke Zettlemoyer and Iz Beltagy and Dirk Groeneveld and Jesse Dodge and Kyle Lo},year={2023},journal={arXiv preprint},}
Package: dolmen.app.content
dolmen.app.contentprovides out-of-the-box utilities for Dolmen
applications content.Getting startedWe import Grok and grok the package:>>> import grok
>>> grok.testing.grok('dolmen.app.content')We create adolmen.contentcontent:>>> import dolmen.content
>>> class Mammoth(dolmen.content.Content):
...    grok.name('A furry thing')
>>> manfred = Mammoth()
>>> manfred.title = u'A nice mammoth'Indexesdolmen.app.contentregisters two indexes to catalog thedolmen.content.IBaseContentcreated inside a Dolmen application:>>> from dolmen.app.content import indexesBase indexesThedolmen.app.content.indexes.BaseIndexeshandles the title and the content type of an IBaseContent content:>>> indexes.BaseIndexes.__grok_indexes__
{'content_type': <grok.index.Field object at ...>, 'title': <grok.index.Text object at ...>}

>>> indexes.BaseIndexes.__grok_indexes__['content_type']._attribute
'__content_type__'Searchable textdolmen.app.contentprovides a simple ‘ISearchableText’
implementation, allowing full text searches. It comes in two parts.The index:>>> indexes.SearchableIndex.__grok_indexes__
{'searchabletext': <grok.index.Text object at ...>}The adapter:>>> from zope.index.text.interfaces import ISearchableText
>>> adapter = ISearchableText(manfred)
>>> adapter.getSearchableText()
u'A nice mammoth'ThumbnailingThanks todolmen.thumbnailer,dolmen.app.contentprovides a base
thumbnailing policy, using ZODB blobs as storage and introducing a new
scale.ScalesLet’s introspect our Miniaturizer component:>>> from dolmen.app.content import thumbnails

>>> thumbnails.BlobMiniaturizer.factory
<class 'dolmen.blob.file.BlobValue'>

>>> print thumbnails.BlobMiniaturizer.scales
{'mini': (250, 250), 'square': (64, 64), 'thumb': (150, 150), 'large': (700, 700), 'small': (128, 128), 'preview': (400, 400)}The new scale, ‘square’, scales down and crops the original image to
provide a square thumbnail. This is done using a IThumbnailer adapter:>>> from dolmen.thumbnailer import IThumbnailer
>>> thumbnails.SquareThumbnailer
<class 'dolmen.app.content.thumbnails.SquareThumbnailer'>
>>> IThumbnailer.implementedBy(thumbnails.SquareThumbnailer)
TrueCreditsAll Dolmen packages are sponsorised by NPAI (http://www.npai.fr)Changelog0.2.1 (2010-02-25)Using PIL 1.1.7 instead of PILwoTK 1.1.6. This prevents having to
declare a third party archive download location.Maintenance cleaning for the release ofGrok 1.1rc1. Code is now
pep8 compliant.0.2 (2010-02-01)Added dolmen.blob include in configure.zcml.Cleaning dependencies. The package is now zope.app free and relies
on the latest Grok evolution.0.1 (2009-11-02)Initial release
Package: dolmen.app.security
dolmen.app.securityis a package providing a collection of basic roles and
permissions for a Dolmen application.>>> import dolmen.app.security
>>> from dolmen.app.security import IDolmenSecurity
>>> from dolmen.app.security.roles import IDolmenRoles
>>> from dolmen.app.security.content import IContentPermissions>>> IDolmenSecurity.extends(IDolmenRoles)
True
>>> IDolmenSecurity.extends(IContentPermissions)
True>>> IDolmenSecurity.providedBy(dolmen.app.security)
TrueContent centric permissions>>> import dolmen.app.security.content>>> IContentPermissions.providedBy(dolmen.app.security.content)
True>>> for name, attr in IContentPermissions.namesAndDescriptions():
...   print "%s: %s" % (name, attr.getDoc())
CanEditContent: Edit content.
CanAddContent: Add content.
CanListContent: List the container content.
CanDeleteContent: Delete content.
CanReviewContent: Review and annotate content.
CanCopyContent: Copy content to the clipboard.
CanPasteContent: Paste content from the clipboard.
CanViewContent: View content.Dolmen application roles>>> import dolmen.app.security.roles>>> IDolmenRoles.providedBy(dolmen.app.security.roles)
True>>> for name, attr in IDolmenRoles.namesAndDescriptions():
...   print "%s: %s" % (name, attr.getDoc())
Member: A basic member.
Contributor: A content contributor.
Owner: The owner of an object.
Reviewer: A content reviewer.Changelog0.1 (2009-10-27)Initial release
Package: dolmen.app.site
Test-Layer:functionalDolmen is a simple application. The packagedolmen.app.siteprovides
an easy way to bootstrap your own applications. It exposes two elements,
the Dolmen object and the IDolmen interface.This creation of Dolmen Application is straightforward:>>> import grokcore.component as grok
>>> from zope.event import notify
>>> from zope.lifecycleevent import ObjectCreatedEvent
>>> from dolmen.app.site import Dolmen, IDolmen

>>> rocks = Dolmen()
>>> notify(ObjectCreatedEvent(rocks))

>>> IDolmen.providedBy(rocks)
TrueIt’s a validzope.componentISite:>>> import zope.component
>>> from zope.component.interfaces import ISite, IPossibleSite
>>> from zope.site.site import LocalSiteManager>>> IPossibleSite.providedBy(rocks)
True>>> ISite.providedBy(rocks)
False>>> site = LocalSiteManager(rocks)
>>> rocks.setSiteManager(site)
>>> rocks.getSiteManager()
<LocalSiteManager ++etc++site>Dolmen is adolmen.contentContainer (readdolmen.contentdocumentation for more information):>>> from dolmen.content import IContent, IContainer
>>> IContent.providedBy(rocks)
True
>>> IContainer.providedBy(rocks)
TrueChangelog0.2 (2011-03-06)Added proper license file and tagging (ZPL).Don’t use z3c.testsetup for tests anymore.Dependencies onGrokhas been dumped. We now depend ongrokcore.site.Cleaned code and dependencies.Stripped down theftesting.zcmlof all the useless statements.0.1 (2009-10-27)Internal release
Package: dolmen.beaker
SessionConfiguration>>> from zope.component import getUtility, queryUtility
>>> from dolmen.beaker.interfaces import ISessionConfig>>> session_config = queryUtility(ISessionConfig)
>>> session_config.get('key')
'beaker.session.id'Initialisationdolmen.beakerinitialize the beaker session when the traversing
starts (traversing the IRootFolder object):>>> from zope.event import notify
>>> import grokcore.component as grok
>>> from zope.publisher.browser import TestRequest
>>> from zope.publisher.interfaces.http import IHTTPRequest
>>> from zope.traversing.interfaces import BeforeTraverseEventFirst we create an instance of the Request:>>> request = TestRequest()
>>> IHTTPRequest.providedBy(request)
TrueWe initalize our Session with the help of the BeforeTraverseEvent.
This event will make an instance of a Session Object in the Request:>>> from zope.component.hooks import getSite
>>> site = getSite()
>>> notify(BeforeTraverseEvent(site, request))We can get the session object using two available adapters:>>> from dolmen.beaker.interfaces import ISession
>>> session = ISession(request)
>>> session.__class__
<class 'beaker.session.SessionObject'>Let’s assign an value to our session and save it:>>> session['foo'] = 'bar'
>>> session.save()
>>> session['foo']
'bar'A new request will not have access to the session without traversing:>>> newrequest = TestRequest()
>>> newrequest.response._cookies
{}

>>> session = ISession(newrequest)
>>> session['foo']
Traceback (most recent call last):
...
KeyError: 'foo'closure and invalidation>>> cookie = request.response._cookies
>>> cookie
{}>>> from zope.publisher.interfaces import EndRequestEvent
>>> notify(EndRequestEvent(site, request))>>> cookie = request.response._cookies
>>> cookie
{'beaker.session.id': {'path': '/', 'samesite': 'Lax', 'value': '...'}}Invalidating>>> import hamcrest
>>> session = ISession(request)
>>> print(session['foo'])
bar>>> session.invalidate() # Or destroy, to get rid of everything
>>> session['foo']
Traceback (most recent call last):
...
KeyError: 'foo'The Zope Session adapter>>> from zope.session.interfaces import ISession as IZopeSession>>> request = TestRequest()
>>> notify(BeforeTraverseEvent(site, request))>>> zsession = IZopeSession(request)
>>> print(zsession)
<dolmen.beaker.session.ZopeSession object at ...>>>> from zope.interface.verify import verifyObject
>>> verifyObject(IZopeSession, zsession)
True>>> data = zsession['my_package']
>>> print(data)
<dolmen.beaker.session.NamespaceSessionData object at ...>>>> data['someitem'] = 'test'
>>> print(data['someitem'])
test>>> data = zsession['some.other.package']
>>> data['info'] = 'Grok !'Changelog0.4 (2020-10-07)Updated MANIFEST0.3 (2020-04-07)Corrected deprecated import and package structure.
Package: dolmen.breadcrumbs
dolmen.breadcrumbsprovides a breadcrumbs navigation for Cromlech.Changelog0.2.1 (2017-03-18)Addedconfigure.zcmlfile to have registrations working.Fixed tests.0.2 (2012-05-09)Updated to use the latest changes incromlech.browser. The API is
unchanged but versions have been pinned down. Please check your versions
before updating.0.1 (2012-01-20)Initial release : forked out ofdolmen.app.breadcrumbs.
Package: dolmen.builtins
dolmen.builtinsprovides a collection of interfaces representing the
commonly used Python built-ins. It aims to make the component
architecture useable with the most basic objects and to defines the
types in order to extend them conveniently.>>> from zope.interface import verify
>>> from dolmen.builtins import interfaces as base>>> macduff = "Tis' a very nice string."
>>> base.IString.providedBy(macduff)
True>>> macbeth = u"Aye, indeed my friend."
>>> base.IUnicode.providedBy(macbeth)
True>>> is_usurper = True
>>> base.IBoolean.providedBy(is_usurper)
True>>> crown = 1
>>> base.INumeric.providedBy(crown)
True>>> king = 0.1
>>> base.INumeric.providedBy(king)
True>>> opposants = 1L
>>> base.INumeric.providedBy(opposants)
TrueMore complex types have more information defined in their interfaces.
It’s the case for iterables and file-like classes.>>> murderers = ('MacBeth', 'Lady MacBeth')
>>> base.ITuple.providedBy(murderers)
True>>> victims = ['Banco', 'Duncan']
>>> base.IList.providedBy(victims)
True>>> thanes = {"Glamis": "MacBeth", "Fife": "MacDuff"}
>>> base.IDict.providedBy(thanes)
True>>> base.IIterable.providedBy(victims)
True
>>> base.IIterable.providedBy(murderers)
True
>>> base.IIterable.providedBy(thanes)
True>>> verify.verifyObject(base.IDict, thanes)
True>>> base.IFile.implementedBy(file)
True
>>> verify.verifyClass(base.IFile, file)
TrueChangelog0.3.1 (2010-02-27)Remove zope.testing dependency.0.3 (2010-02-27)Cleaned code. It’s now pep8 compliant.Cleaned dependencies. We no longer depend on zope.app.0.2 (2009-10-16)Addedfileto the marked built-ins, using a detailed interface.0.1 (2009-10-15)Initial release
Package: dolmen.collection
Dolmen collection offers a collection implementation which permits to
manipulate named elements: arrange order, extract part of the collection,
enforce a type of component and add new ones easily.It originated from zeam.form where it was used to implement forms as
collections of fields, actions and widgets.Changes0.5 (2020-03-30)Moved to python 3.6+ only.Added requirements.txt for pip installation.0.4Reduced the dependencies by removing the unnecessary call ofzope.componentin the adaptation of the factory.0.3 (2011-06-13)Added behavior to the collection allowing the override or ignore of the
components.0.2 (2011-04-14)Updated components names for entry points.0.1 (2011-04-13)Initial release.
Package: dolmen.field
Thedolmen.fieldpackage provides additional generic fields to
extendzope.schema.Configuration fieldsThe configuration fields are used for description and constraints, to
impose a consistent schema and data model.GlobalClassGlobalClassis a field requiring a valid Interface class. It can be
used for factories and other components that need to store an
interface with the possibility to check and verify the consistency of
the given value:>>> from zope.interface import Interface
>>> from dolmen.field import GlobalClass, IClassField

>>> class IDummy(Interface):
...     def shall_fail():
...         pass

>>> class Dummy(object):
...     passOf course, it must be a valid interface:>>> field = GlobalClass(
...             title=u'a class field',
...             schema=object
...             )
Traceback (most recent call last):
...
WrongType: <type 'object'> is not a valid InterfaceThe interface will be used to validate the class:>>> field = GlobalClass(
...             title=u'a class field',
...             schema=IDummy
...             )

>>> field.schema
<InterfaceClass dolmen.field.tests.IDummy>
>>> IClassField.providedBy(field)
TrueWe make sure theinterfaceattribute is not touched:>>> print field.interface
NoneIn a first time, we make sure the class implements the needed interface:>>> field._validate(Dummy)
Traceback (most recent call last):
...
ConstraintNotSatisfied: <class 'dolmen.field.tests.Dummy'> does not implement <InterfaceClass dolmen.field.tests.IDummy>

>>> from zope.interface import classImplements
>>> classImplements(Dummy, IDummy)
>>> IDummy.implementedBy(Dummy)
TrueThen the interface acts as a strict validator:>>> field._validate(Dummy)
Traceback (most recent call last):
...
BrokenImplementation: An object has failed to implement interface <InterfaceClass dolmen.field.tests.IDummy>
<BLANKLINE>
The shall_fail attribute was not provided.
<BLANKLINE>

>>> Dummy.shall_fail = lambda x: u'not'
>>> field._validate(Dummy) is None
TrueChangelog0.3 (2009-10-25)Refactored the field to use the ‘schema’ attribute instead of
‘interface’. The attribute name ‘interface’ was conflicting with
other fields manipulating packages.Now using a README.txt instead of inline doctests, for more
readability.0.2 (2009-06-09)Initial Pypi release0.1 (2009-05-03)Internal release
Package: dolmen.menu
dolmen.menuaims to provide the most flexible and explicit way to
create and manage menus and their entries with Grok.You have to know that…dolmen.menuonly works on Python 2.6.dolmen.menudoes not support sub-menus, out of the box. The
implementation is left to your discretion.Componentsdolmen.menuprovides two components:Menu : the menu implementation is based on the zope “content
provider” notion and is using thegrokcore.viewletpackage.Entry: a menu item is called an entry. It’s a multi adapter that is
registered for a Menu component.ExamplesA menu component:>>> from dolmen import menu

>>> class MyMenu(menu.Menu):
...     menu.title('My nice menu')A menu entry:>>> class MyMenuEntry(menu.Entry):
...     menu.order(1)
...     menu.name('a_direct_entry')
...     menu.title('My Entry')
...     menu.menu(MyMenu)RegistrationIn order to use base Grok component as menu entries, we get two
registration ways.class decoratorA class decorator allows you to decorate any class, in order to
register it as a menu entry:>>> import grokcore.view

>>> @menu.menuentry(MyMenu)
... class TestEntry(grokcore.view.View):
...    def render(self):
...        return u"A simple entry"Module level martian directiveA martian directive allows you register classes you can’t decorate
(from a foreign package, for instance), explicitly:>>> class SomeView(grokcore.view.View):
...    def render(self):
...        return u"I'm a view and I want to be a menu entry"

>>> menu.global_menuentry(SomeView, MyMenu, order=2)Changes0.5 (2011-01-31)dolmen.menunow works on the latestmartian. The treatment
of the decorator has been deferred by the use of a module
annotation. The registration is now done by a global grokker.0.4 (2010-12-24)It is now possible to define a menu context, that is different of
the rendering context. This allow delegated rendering, such as Root
menus or other object-specific menus.0.3 (2010-06-04)Corrected a bug where ‘zope.Public’ was evaluated as a “normal”
permission. Now, the manager converts the permission into a
CheckerPublic marker.thefiltermethod is now a generator.0.2 (2010-03-05)Changed the way to fetch the menu CSS id.0.1 (2010-03-03)Initial release.
Package: dolmenproject
Script that creates a Dolmen project directory, installs Grok, the Grok Toolkit and the Zope Toolkit and sets up a complete skeleton for a new Dolmen application.ContentsDescriptionChanges1.0a3 (2011-02-19)1.0a2 (2011-01-11)1.0a1 (2011-01-11)Descriptiondolmenproject provides an easy way to get started with aDolmenweb application.
It’s an extension of grokproject package.This documentation is adapted from grokproject README.txt.
And so is the most of the code :). Thanks to grokproject developers!Simply installdolmenproject:$ easy_install dolmenprojectIf you have an old version ofdolmenprojectinstalled then you can
upgrade doing:$ easy_install -U dolmenprojectAfterwards you can run thedolmenprojectscript with the name of the
project you’d like to create as an argument:$ dolmenproject DruidicRitual
... many lines of output hereThis will not only create a project area for you to work in, it will
also download and install Grok and its dependencies.After the project area has been created successfully, you will find an empty
Python package “skeleton” in thesrcdirectory in which you can place the
code for your web application.To start the application server, execute:$ cd DruidicRitual
$ bin/paster serve parts/etc/deploy.iniStart/stop it in daemon mode:$ bin/daemon start/stopThere is also an Ajax enabled debugger (point your browser tohttp://localhost:8080/@@login.htmlwhen using this):$ bin/paster serve parts/etc/debug.iniTo start the interactive debugger prompt:$ bin/interactive_debuggerTo run an ad-hoc Python script against a full setup application:$ bin/interactive_debugger [name_of_python_script].pyPython scripts run this way, will have access to aroot,debugger, and
anappobject avaible for “interacting” with the application environment.For those who know paster:grokprojectis just a wrapper around a
paster template.  So instead of running thegrokprojectcommand,
you can also run:$ paster create -t dolmen DruidicRitualAll configuration files used for running Grok can be found in theparts/etc/directory of your project. These configuration files are
generated automatically from the templates inetc/on eachbuildoutrun. To modify the configuration files edit the approriate templates inetc/and rerunbuildoutafterwards:$ bin/buildoutThis will rebuild the files inparts/etc/.Changes1.0a3 (2011-02-19)Compatible with grokproject 2.5.Now search latest Dolmen version athttp://www.dolmen-project.org/kgs/current1.0a2 (2011-01-11)Include .zcml_tmpl files in the release.Define “sources-dir = dev” in generated sources.cfg.1.0a1 (2011-01-11)Initial version, based on grokproject 2.3.
Package: dolmen.sqlcontainer
dolmen.sqlcontaineris an addon for cromlech and sqlalchemy. It provides
a “containership” abstraction, to work transparently with a hierarchy of
objects coming from an SQL database.Changelog0.3 (2018-12-20)Added slicing on the containerChanged the Container init to take a session_getter.0.2 (2014-03-14)Add a MANIFEST.INI0.1 (2014-03-13)Initial release
Package: dolmen.widget.tinymce
dolmen.widget.tinymceis a package that provides a useable and
pluggable way to render a text field as a WYSIWG editor in azeam.formForm.ExampleWe are going to develop here a small example, to demonstrate the use
ofdolmen.widget.tinymce. First, we need to create a model content with
a text field:>>> import grokcore.component as grok
>>> from zope.interface import Interface
>>> from zope.schema import Text
>>> from zope.schema.fieldproperty import FieldProperty

>>> class ICave(Interface):
...   paintings = Text(title=u'Description of the cave paintings')

>>> class Grotto(grok.Context):
...   paintings = FieldProperty(ICave['paintings'])We have now a model that defines a text field. We want to edit/view
this content, using a rich editor, allowing to input rich text and to
display it as valid HTML. To do so, we define a form:>>> from zeam.form.ztk import Form, Fields>>> class EditCave(Form):
...    grok.name('edit')
...    grok.context(ICave)
...    ignoreContent = False
...    fields = Fields(ICave)>>> grok.testing.grok_component('edit', EditCave)
TrueAt this point, if we instanciate the form, we have a normal
rendering:>>> from zope.publisher.browser import TestRequest

>>> homecave = Grotto()
>>> request = TestRequest()

>>> form = EditCave(homecave, request)
>>> form.updateWidgets()

>>> print form.fieldWidgets.get('form.field.paintings').render()
<textarea id="form-field-paintings" name="form.field.paintings" class="field field-text field-required" cols="80" rows="5"></textarea>To get the tinyMCE widget, you simply need to use the “mode” of the
field, to indicate what you want to render:>>> from dolmen.widget.tinymce import TINYMCE_INPUT

>>> form = EditCave(homecave, request)
>>> form.fields['paintings'].mode = TINYMCE_INPUT
>>> form.updateWidgets()

>>> print form.fieldWidgets.get('form.field.paintings').render()
<script type="text/javascript">
   $(document).ready(function(){
      $('textarea[name="form.field.paintings"]').tinymce();
   });
</script>
<textarea id="form-field-paintings"
          name="form.field.paintings"
          class="field field-text field-required"
          cols="80" rows="5"></textarea>The modes can be ‘tinymce.input’ for an input widget and
‘tinymce.display’ to display the value as valid html:>>> from dolmen.widget.tinymce import TINYMCE_DISPLAY
>>> homecave.paintings = u"<h1>Very nice paintings</h1><p>Mammoth</p>"

>>> form = EditCave(homecave, request)
>>> form.fields['paintings'].mode = TINYMCE_DISPLAY
>>> form.updateWidgets()

>>> print form.fieldWidgets.get('form.field.paintings').render()
<div id="form-field-paintings" name="form.field.paintings"
     class="field"><h1>Very nice paintings</h1><p>Mammoth</p></div>Changelog1.0b3 (2011-01-18)Corrected entry points and inclusion path.1.0b2 (2011-01-13)Now we’re usingfanstatic.1.0b1 (2010-07-06)Tuned the menus of the TinyMCE widget, in order to removed useless
options.The package is now using the latest changes in thezeam.formcore, such as the ModeMarker, in order to get sane mode definitions.1.0a1 (2010-06-06)Initial release