dataesr/scientific-paragraphs-categorization huggingface dataset:
In response, he wrote a plotting library from scratch, and Matplotlib 0. 1 was released April 2003  39. As STScI continued to use Python for an increasingly large portion of the Hubble Space Telescope data analysis pipeline, they encountered problems with the Python numerical array container. Numeric, the original array package, was suitable for small arrays, but not for the large images processed by STScI. With the Numeric maintainer's blessing, the decision was made to write NumArray  40  , a library that could handle data on a larger scale. Unfortunately, NumArray proved inefficient for small arrays, presenting the community with a rather unfortunate choice. In 2005, Travis Oliphant combined the best elements of Numeric and NumArray, thereby solving the dilemma and paving the way for Python to become a very significant player in the Data Science movement.
SciPy Matures By the middle to late 2000s, SciPy was starting to mature after a long phase of significant growth and adoption. The informal workshops grew into international conferences with many hundreds of attendees. Special issues were organized and published in a leading scientific journal  42  , and the scope of the SciPy library narrowed, while the breadth of the ecosystem grew through a new type of auxiliary package: the scikit 55. The tooling, development, documentation, and release processes became more professional. SciPy was expanded carefully, with the patience affordable in open source projects and via best practices common in industry  56. Early versions of SciPy had minimal documentation, but this began to change with the 2006 release of a Guide to NumPy 1. In 2007, Sphinx 57 made it possible to render hypertext and PDF-formatted documents automatically from plain text (docstrings) interspersed with Python code, and in 2008, pydocweb 58 enabled collaborative documentation development in a wiki-like fashion.
The SciPy Documentation Project  59, 60  used these tools to complete documentation of SciPy's user-facing functionality: offering t-shirts to contributors from around the world in exchange for high-quality text, it collected contributions from over 75 people to produce an 884 page manual  61. Since then, SciPy has remained committed to maintaining high-quality documentation as part of the normal development cycle. In the early workshops, recurrent topics reflected the state of development, with emphasis being placed on the underlying array package, plotting, parallel processing, acceleration / wrapping, and user interfaces. By 2004, a significant shift occurred towards application of SciPy to scientific problems. The event also started to draw in more keynote speakers from outside the community, such as Guido van Rossum (creator of Python, 2006), Ivan Krstić (One Laptop per Child, 2007), Alex Martelli (Google, 2008), and Peter Norvig (Google Research, 2009).
The SciPy conference went from being a small gathering of core developers to a multi-location event with increased funding, a published proceedings, and scholarships for attending students. By 2010, the US SciPy conference had multiple tracks, and satellite conferences were being organized by volunteers elsewhere, such as EuroSciPy (2008-) and SciPy India (2009-). Special sessions and minisymposia dedicated to scientific Python began appearing at many other events. For example, a three-part minisymposium organized for CSE 2009 was featured in SIAM News  62. In 2007, Python had a strong enough presence in science and engineering that the editors of IEEE Computing in Science and Engineering (CiSE) solicited a special issue  42  , edited by Paul Dubois. However, Python was still sufficiently niche that the average reader would need additional information in order to decide whether it would "be useful in their own work.
" The follow-up CiSE March/April 2011 Python for Scientists and Engineers special issue  63  focused more on the core parts of the scientific Python ecosystem  64  including NumPy 2 , Cython  65  , and Mayavi 66. Python became so pervasive that journals began publishing domain-specific special issues. For example, in 2015 Frontiers in Neuroinformatics published a collection of 25 articles-covering topics including modeling and simulation, data collection, electrophysiology, visualization, as well as stimulus generation and presentation-called Python in Neuroscience  67. In 2012, Perry Greenfield, John Hunter, Jarrod Millman, Travis Oliphant, and Fernando Pérez founded NumFOCUS  68  , a 501(c)3 public charity with a mission "to promote sustainable high-level programming languages, open code development, and reproducible scientific research.
Architecture and implementation choices Project scope SciPy provides fundamental algorithms for scientific computing. The breadth of its scope was derived from the Guide to Available Mathematical Software classification system (GAMS  70  ). In areas that move relatively slowly, e. g. linear algebra, SciPy aims to provide complete coverage. In other areas it aims to provide fundamental building blocks while interacting well with other packages specialized in that area. For example, SciPy provides what one expects to find in a statistics textbook (probability distributions, hypothesis tests, frequency statistics, correlation functions, and more), while Statsmodels 71 provides more advanced statistical estimators and inference methods; scikit-learn 3 covers machine learning; and PyMC3 72 , emcee  73  and PyStan  74  cover Bayesian statistics and probabilistic modeling.
scikit-image  4  provides image processing capabilities beyond SciPy's ndimage, SymPy 75 provides a Python interface for symbolic computation, and while sparse. csgraph or spatial offers basic tools for working with graphs and networks compared to a more specialized Python library like NetworkX  76. We use the following criteria to determine whether or not to include new functionality in SciPy: • The algorithm is of relevance to multiple fields of science. • The algorithm is demonstrably important. For example, it is classic enough to be included in textbooks, or it is based on a peer-reviewed article which has a significant number of citations. In terms of software systems and architecture, SciPy's scope matches NumPy's: algorithms for in-memory computing on single machines, with support for a wide range of data types and process architectures.
Language choices According to analysis using the linguist library  83  , SciPy is approximately 50% Python, 25% Fortran, 20% C, 3% Cython, and 2% C++, with a dash of T E X, MATLAB, shell script, and Make. The distribution of secondary programming languages in SciPy is a compromise between a powerful, performance-enhancing language that interacts well with Python (i. e. Cython) and the usage of languages (and their libraries) that have proven reliable and performant over many decades. Fortran, despite its age, is still a high-performance scientific programming language with continued contemporary usage  84. Thus, we wrap the following excellent, field-tested Fortran libraries in order to provide Python convenience while benefiting from their performance: FFTPACK  85, 86  , ODEPACK  23  , QUADPACK  24  , FITPACK  78  , ODRPACK  82  , MINPACK  25  , ARPACK  87  , ALGORITHM 644  88  , and CDFLIB  89.
Rounding out the top three languages in SciPy is C, which is also extremely well-established over several decades  90  of scientific computing. The C libraries that we wrap in SciPy include trlib  91  , SuperLU  92, 93  , Qhull  94  , and Cephes  95. Cython has been described as a creole language that mixes the best parts of Python and lower-level C / C++ paradigms  65. We often use Cython as a glue between well-established, low-level scientific computing libraries written in C / C++ and the Python interface offered by SciPy. We also use Cython to enable performance enhancements in Python code, especially for cases where heavily used inner loops benefit from a compiled code with static typing. For implementing new functionality, Python is the still the language of choice. If Python performance is an issue, then we prefer the use of Cython followed by C, C++, or Fortran (in that order).
The main motivation for this is maintainability: Cython has the highest abstraction level and most Python developers will understand it. C is also widely known, and easier for the current core development team to manage than C++ and especially Fortran. The position that SciPy occupies near the foundation of the scientific Python ecosystem is such that adoption of new languages or major dependencies is generally unlikely-our choices are strongly driven by long-term stability. GPU acceleration, new transpiling libraries, and the latest JIT compilation approaches (i. e. , Numba  96  ) are very powerful, but currently fall outside the remit of the main SciPy library. That said, we have recently increased our efforts to support compatibility with some of these options, and having our full test suite pass with the PyPy JIT compiler  97  is now a requirement in our development workflow.
API and ABI evolution The application programming interface (API) for SciPy consists of approximately 1,500 functions and classes. Our policy for evolving the API over time is that new functionality can be added, while removing or changing existing functionality can only be done if the benefits exceed the (often significant) costs to users, and only after giving clear deprecation warnings to those users for at least one year. In general, we encourage changes that improve clarity in the API of the library but strongly discourage breaking backwards compatibility given our position near the base of the scientific Python computing stack. In addition to the Python API, SciPy has C and Cython interfaces. Therefore, we have to also consider the application binary interface (ABI). This ABI has been stable for a long time, and we aim to evolve it only in a backwards compatible way.
This routine is not limited to the conventional L2 (Euclidean) norm but supports any Minkowski p-norm between 1 and infinity. By default, the returned data structure is a Dictionary Of Keys (DOK) based sparse matrix, which is very efficient for matrix construction. This hashing approach to sparse matrix assembly can be 7 times faster than constructing with CSR format  101  , and the C++ level sparse matrix construction releases the Python GIL for increased performance. Once the matrix is constructed, distance value retrieval has an amortized constant time complexity  102  , and the DOK structure can be efficiently converted to a CSR, CSC, or COO matrix to allow for speedy arithmetic operations. In 2015 the cKDTree dual tree counting algorithm  103  was enhanced to support weights  104  , which are essential in many scientific applications, e.
LowLevelCallable As of SciPy version 0. 19, it is possible for users to wrap low-level functions in a scipy. LowLevelCallable object that reduces the overhead of calling compiled C functions, such as those generated using numba or Cython, directly from Python. Supported low-level functions include PyCapsule objects, ctypes function pointers, and cffi function pointers. Furthermore, it is possible to generate a low-level callback function automatically from a Cython module using scipy. LowLevelCallable. from cython.
Cython bindings for BLAS, LAPACK, and special SciPy has provided special functions and leveraged BLAS and LAPACK  106  routines for many years. SciPy now additionally includes Cython  65  wrappers for many BLAS and LAPACK routines (added in 2015) and the special functions provided in the scipy. special subpackage (added in 2016). These Cython wrappers are available in scipy. linalg. cython blas, scipy. linalg. cython lapack, and scipy. special. cython special respectively. When writing algorithms in Cython, it is typically more efficient to call directly into the libraries SciPy wraps rather than indirectly, using SciPy's Python APIs. These low-level interfaces for Cython can also be used outside of the SciPy codebase to gain access to the functions in the wrapped libraries while avoiding the overhead of Python function calls.
Linear Optimization A new interior-point optimizer for continuous linear programming problems, linprog with method='interior-point', was released with SciPy 1. 0. Implementing the core algorithm of the commercial solver MOSEK  108  , it solves all of the 90+ NETLIB LP benchmark problems 109 tested. Unlike some interior point methods, this homogeneous self-dual formulation provides certificates of infeasibility or unboundedness as appropriate. A presolve routine  110  solves trivial problems and otherwise performs problem simplifications, such as bound tightening and removal of fixed variables, and one of several routines for eliminating redundant equality constraints is automatically chosen to reduce the chance of numerical difficulties caused by singular matrices. Although the main solver implementation is pure Python, end-to-end sparse matrix support and heavy use of SciPy's compiled linear system solvers-often for the same system with multiple right hand sides due to the predictor-corrector approach-provide speed sufficient for problems with tens of thousands of variables and constraints.
Test suite The SciPy test suite is orchestrated by a continuous integration matrix that includes POSIX and Windows (32/64-bit) platforms managed by Travis CI and AppVeyor, respectively. Our tests cover Python versions 2. 7, 3. 4, 3. 5, 3. 6, and include code linting with pyflakes and pycodestyle. There are more than 13, 000 unit tests in the test suite, which is written for usage with the pytest 141 framework. Test coverage at the SciPy 1. 0 release point (Figure  3 ) was at 87% for Python code according to pytest-cov  142  and 45% for compiled (C, C++, and Fortran) code according to gcov  143. Documentation for the code is automatically built and published by the CircleCI service to facilitate evaluation of documentation changes / integrity. Our full test suite also passes with PyPy3 Coverage analysis is performed automatically in our continuous integration suite using the pytest-cov and gcov libraries for Python and compiled code, respectively.
Downstream projects The scientific Python ecosystem includes many examples of domain-specific software libraries building on top of SciPy features and then returning to the base SciPy library to suggest and even implement improvements. For example, there are common contributors to the SciPy and Astropy core libraries  149  , and what works well for one of the codebases, infrastructures, or communities is often transferred in some form to the other. At the codebase level, the binned statistic functionality is one such cross-project contribution: it was initially developed in an Astropy-affiliated package and then placed in SciPy afterwards. In this perspective, SciPy serves as a catalyst for cross-fertilisation throughout the Python scientific computing community.
Discussion SciPy has a strong developer community and a massive user base. GitHub traffic metrics report roughly 20,000 unique visitors to the source website between May 14 and May 27, 2018, with 721 unique copies ("clones") of the codebase over that roughly two-week period. The developer community at that time consisted of 610 unique contributors of source code, with > 19, 000 commits accepted into the codebase (GitHub page data). From the user side, there were 13,096,468 downloads of SciPy from the Python Packaging Index (PyPI)  150  and 5,776,017 via the default channel of the conda 151 package manager during the year 2017. These numbers establish a lower bound on the total number of downloads by users given that PyPI and conda are only two of several popular methods for installing SciPy.
The SciPy website  152  , which has been the default citation in the absence of a peer-reviewed paper, has been cited > 3, 000 times  153. Some of the most prominent usages of or demonstrations of credibility for SciPy include the LIGO / Virgo scientific collaboration that lead to the observation of gravitational waves  154  , the fact that SciPy is shipped directly with macOS and in the Intel Distribution for Python  155  , and that SciPy is used by 47% of all machine learning projects on GitHub  156. Nevertheless, SciPy continually strives to improve. The SciPy Roadmap  157, 158  , summarized in Table  2 , is a continuouslyupdated document maintained by the community that describes some of the major directions for improvement for the project, as well as specific limitations and matters that require assistance moving forward.
In addition to the items on the roadmap, we are still working to increase the number of SciPy usage tutorials beyond our current 15 section offering  159. Also, the low-level Cython code in our library (which interacts with C-level code and exposes it for Python usage) could use some measure of modernization, including migration to typed memoryviews to handle NumPy arrays. A problem faced by many open source projects is attracting and retaining developers. While it is normal for some individuals to contribute to a project for a while and then move on, too much turnover can result in the loss of institutional memory, leading to mistakes of the past being repeated, APIs of new code becoming inconsistent with the old code, and a drifting project scope. We are fortunate that the SciPy project continues to attract enthusiastic and competent new developers while maintaining the involvement of a small but dedicated "old guard".
There are contributors who were present in the early years of the project who still contribute to discussions of bug reports and reviews of new code contributions. Our BDFL has been with the project for more than 10 years and is still actively contributing code, and the head of our Steering Council, who also acts as a general manager, is approaching his eleventh anniversary. An additional half dozen or so active developers have been contributing steadily for five or more years. The combination of a committed old guard and a host of new contributors ensures that SciPy will continue to grow while maintaining a high level of quality.
More details are given in Section IV (algorithm engineering). As a result, our execution time on real-world networks compares favorably with the most recent state-of-theart approaches for truss decomposition, coming from the GraphChallenge  [23]  and other papers on large real-world and synthetic graphs, as detailed in Section VI, with only one solution (KM17) having slightly less space requirement per edge. We show that our algorithm outperforms all similar algorithms on large networks (i. e. , with millions of edges or more) by up to orders of magnitude. Moreover, using the USD cost per hour of equivalent infrastructure in Google Compute Engine (https://cloud. google. com/compute/), our algorithm results in significant financial savings, such as processing the largest amount of edges per USD cent, except for one dataset, relative to other algorithms.
If M = ∞, FMT finds the max-truss in G. We believe that this is a further important contribution of this paper for the following reasons. • The problem of finding the max-truss in G or its trussness t G is quite natural. It has a similar flavor to that of finding the largest clique and the maximum k-core, or just their size, when computing all cliques or the k-core decomposition: these structures give important information for network analysis and as such their efficient computation is widely studied  [14] ,  [15] ,  [29] ,  [32]  (in particular, the maximum k for a k-core exists is also known as the degeneracy and is used as a sparsity measure  [15] ,  [16] ). • We comment on real-world datasets in Section III, where the max-truss provides often meaningful communities. This has a good foundation in Section IV, where we link for the first time the trussness t G to the densest subgraphs of G in terms of the number of triangles per edge (see Theorem 1, which is an extension of the Nash-Williams's theorem  [1]  to the trussness).
In this scenario, considering the maxtrusses specializes the query for the more cohesive subgraphs (i. e. for the largest feasible k), when memory is limited. Depending on the network's topology, the communities found, as mentioned above, can be further analyzed and refined by inspection. In order to illustrate this task, we consider four datasets taken from LAW (http://law. di. unimi. it/). Their size and statistics are shown in Table  1  (we refer the reader to Section VI for performance analysis of our approach). Web snapshot: gsh-2015-host This is the host graph of the graph gsh-2015, which is a large snapshot of the web (988M nodes and 33G edges) taken in 2015 by BUbiNG  [6]  starting from the site http://europa. eu/ without any domain restriction. In the host graph, pages with the same domain name (host) are collapsed; the maximum number of pages per host was set to 100, to find a large number of hosts, and the resulting host graph is composed of 68M nodes and 1.
Movie actors: hollywood-2011 In this graph extracted from the Internet Movie Database (IMDb) vertices are actors, and two actors are joined by an edge whenever they appeared in a movie together. The maxtruss corresponds to a community of 1298 nodes, which is a clique of the actors starring in the movie "Around the World in Eighty Days (1956)". The movie, based on the homonym novel by Jules Verne, follows Victorian Englishman Phileas Fogg (David Niven) in several locations around the globe, and thus stars a significant amount of background actors. Notably, it seems that many of these were registered in the IMDb database, resulting in this large clique. Decreasing k to k = 1000 we obtain a connected k-truss of size 3529, composed by 3 cliques of similar size: the above clique, one clique of actors active in the 80s (mostly Spanish), and one containing Hollywood superstars (e.
Once again, it is worth observing the presence, in each component, of many maximal cliques which are difficult to aggregate into a single community, while connected k-trusses seems to isolate and aggregate the data quite accurately. 1 u ← 0, ← 0 2 while |E(G)| > M do // Part I 3 T G ← 1 3 e∈E(G) sup G [e] 4 ← max( , minsup(G), T G |E(G)| ) 5 u ← max(u, minsup(G), r T G |E(G)| ) 6 Q ← {e ∈ E(G) : sup G [e] ≤ u} // edges to delete 7 G ← G \ Q // remove Q's edges from E(G) 8 t ← u 9 while E(G) = ∅ do // Part II 10 Q ← {e ∈ E(G) : sup G [e] ≤ t} // edges to delete 11 G ← G \ Q // remove Q's edges from E(G), update T G 12 if t > u then foreach e ∈ Q do truss[e] ← t + 2 13 t ← max(t, minsup(G)) 14 if t > u then // values > u + 2 in truss[] are exact 15 return truss[], t G = t + 2 16 else // + 2 ≤ t G ≤ r( + 2) 17 r ← 1, u ← , G ← input graph // reset parameters 18 goto step 2 Bibliographic database: dblp-2011 The graph is a 2011 snapshot of the scientific bibliography service DBLP, from which an undirected collaboration network can be extracted: each vertex represents a scientist and two vertices are connected if they co-authored an article.
State-of-the-art algorithms We will compare our algorithm FMT-dec for finding the truss decomposition with the following ones, which are the most recent state of the art algorithms for k-truss computation. 1) SL+17: a parallel shared-memory algorithm proposed in  [28] , finalist in the 2017 GraphChallenge  [23]. 2) KM17: proposed in  [19]  and won a student innovation award in the same challenge. The code has been downloaded from https://github. com/humayunk1/PKT. 3) WG+18: serial algorithm in  [36] , designed to process large graphs on consumer-grade hardware using the WebGraph framework  [7]. Code kindly provided by the authors  8. 4) D18: the best performing algorithm proposed in  [13] , that is the "highly optimized", parallel implementation in C of the all k-truss algorithm. Code kindly provided by the author.