fozziethebeat/S-Space — pull request #20 by davidjurgens (2012-08-21 06:43:35+00:00)
Title: Star sem 2012

Finally added the ability to project new documents into the LSA/LSI document space. Lots and lots of other minor fixes and adjustments to make this work and/or debuggable.

modified: src/main/java/edu/ucla/sspace/basis/BasisMapping.java

- Fixed javadoc typo

modified: src/main/java/edu/ucla/sspace/common/GenericTermDocumentVectorSpace.java

- Sped up the token counting with a Counter class

modified: src/main/java/edu/ucla/sspace/common/Similarity.java

- Added a new version of Spearman's rho that accounts for ties, as reported in the Nonparametric Statistics handbook. However, this new version doesn't seem to agree with existing results, despite matching the examples in the book...

modified: src/main/java/edu/ucla/sspace/common/Statistics.java

- Added support for getting the mode of a sequence

modified: src/main/java/edu/ucla/sspace/graph/AbstractGraph.java

- Added support for pre-allocating the number of vertices and edges in a graph
to avoid having to rehash large graphs

modified: src/main/java/edu/ucla/sspace/graph/ChineseWhispersClustering.java

- Fixed some bugs (probably?)

- Pushed primitive shuffle to PrimitiveCollections

new file: src/main/java/edu/ucla/sspace/graph/CompactSparseTypedEdgeSet.java

- Alternate implementation of edge sets that is vertex pivoted instead of
type-pivoted. Not sure how to expose this as a configurable paramater yet.

modified: src/main/java/edu/ucla/sspace/graph/SparseTypedEdgeSet.java

- Reworked to be type-pivoted

modified: src/main/java/edu/ucla/sspace/graph/SparseUndirectedEdgeSet.java

- Added support for preallocating space for new edges

modified: src/main/java/edu/ucla/sspace/graph/SparseUndirectedGraph.java

- Added support for preallocating space based on the expected size of the graph

new file: src/main/java/edu/ucla/sspace/graph/io/PajekWriter.java

- Added minimal support for writing in Pajek format

modified: src/main/java/edu/ucla/sspace/lsa/LatentSemanticAnalysis.java

- Lots of fun to tweaks to support new document projection

- Tightened the class constructor to only allow SVD dimensionality reductions,
since this is LSA proper.

modified: src/main/java/edu/ucla/sspace/mains/LSAMain.java

- Reworked to use the new SingularValueDecomposition interface

modified: src/main/java/edu/ucla/sspace/mains/PLSAMain.java

- NOTE: this code is now broken :( We probably need to have a new middle
abstract class between GenericTermDocumentSpace and LSA/PLSA

modified: src/main/java/edu/ucla/sspace/matrix/AbstractMatrix.java

- Added a useful toString()

- Fixed equals() which was broken but for still unexplained reasons.

modified: src/main/java/edu/ucla/sspace/matrix/ArrayMatrix.java
modified: src/main/java/edu/ucla/sspace/matrix/DiagonalMatrix.java
modified: src/main/java/edu/ucla/sspace/matrix/GrowingSparseMatrix.java
modified: src/main/java/edu/ucla/sspace/matrix/RowMaskedMatrix.java
modified: src/main/java/edu/ucla/sspace/matrix/TransposedMatrix.java

- Now extends AbstractMatrix

modified: src/main/java/edu/ucla/sspace/matrix/BaseTransform.java
modified: src/main/java/edu/ucla/sspace/matrix/CorrelationTransform.java
modified: src/main/java/edu/ucla/sspace/matrix/GlobalTransform.java
modified: src/main/java/edu/ucla/sspace/matrix/LogEntropyTransform.java
modified: src/main/java/edu/ucla/sspace/matrix/LogLikelihoodTransform.java
modified: src/main/java/edu/ucla/sspace/matrix/PointWiseMutualInformationTransform.java
modified: src/main/java/edu/ucla/sspace/matrix/RowMagnitudeTransform.java
modified: src/main/java/edu/ucla/sspace/matrix/TfIdfDocStripedTransform.java
modified: src/main/java/edu/ucla/sspace/matrix/TfIdfTransform.java
modified: src/main/java/edu/ucla/sspace/matrix/Transform.java

- Now supports applying the transform to new documents (column vectors) that
weren't in the original data

modified: src/main/java/edu/ucla/sspace/matrix/NoTransform.java

- Now properly supports GlobalTransform so it can be used in LSA

modified: src/main/java/edu/ucla/sspace/matrix/Matrices.java

- Better error message

modified: src/main/java/edu/ucla/sspace/matrix/SVD.java

- Updated to use the new SingularValueDecomposition interfaces

modified: src/main/java/edu/ucla/sspace/matrix/factorization/AbstractSvd.java
new file: src/main/java/edu/ucla/sspace/matrix/factorization/SingularValueDecomposition.java
modified: src/main/java/edu/ucla/sspace/matrix/factorization/SingularValueDecompositionLibC.java
modified: src/main/java/edu/ucla/sspace/matrix/factorization/SingularValueDecompositionLibJ.java
modified: src/main/java/edu/ucla/sspace/matrix/factorization/SingularValueDecompositionMatlab.java
modified: src/main/java/edu/ucla/sspace/matrix/factorization/SingularValueDecompositionOctave.java

- Added new interface for the SVD, which supports returning the U, S, and V
decomposed matrixes and not their premultiplied form

modified: src/main/java/edu/ucla/sspace/text/WordIterator.java

- Removed private protection on the current line so subclass could inspect it

modified: src/main/java/edu/ucla/sspace/tools/IterativeBigramExtractor.java

- Fixed typo in command line options

modified: src/main/java/edu/ucla/sspace/util/KrippendorffsAlpha.java

- Added support for computing alpha using the Interval level of measuremnt

modified: src/main/java/edu/ucla/sspace/util/ObjectCounter.java

- Fixed bug in min()/max()

new file: src/main/java/edu/ucla/sspace/util/Triple.java

- Yet another tuple object

new file: src/main/java/edu/ucla/sspace/util/primitive/DoublePair.java

- Primimite pair to match IntPair

modified: src/main/java/edu/ucla/sspace/util/primitive/IntPair.java

- Fixed formatting

modified: src/main/java/edu/ucla/sspace/util/primitive/PrimitiveCollections.java

- Added the ability to shuffle an int[]

modified: src/main/java/edu/ucla/sspace/vector/SparseHashDoubleVector.java

- Now extends AbstractDoubleVector

modified: src/main/java/edu/ucla/sspace/vector/SparseHashIntegerVector.java

- Fixed performance bug where the magnitude was not cached

modified: src/test/java/edu/ucla/sspace/common/SimilarityTest.java

- Added more unit tests for Spearman's rho

new file: src/test/java/edu/ucla/sspace/graph/CompactSparseTypedEdgeSetTests.java

- New unit tests

modified: src/test/java/edu/ucla/sspace/graph/SparseTypedEdgeSetTests.java

- More unit tests

new file: src/test/java/edu/ucla/sspace/lsa/LatentSemanticAnalysisTest.java

- Tested project() based on worked example on website

modified: src/test/java/edu/ucla/sspace/matrix/ArrayMatrixTests.java

- Added test for matrix equality

modified: src/test/java/edu/ucla/sspace/matrix/FileTransformUtil.java

- Added test for new transform behavior

modified: src/test/java/edu/ucla/sspace/matrix/RowMaskedMatrixTest.java

- Added weird debug to try to figure out what was going with ArrayMatrix.equal()

new file: src/test/java/edu/ucla/sspace/util/KrippendorffsAlphaTest.java

- Added unit tests for some of Alpha's possible computations

modified: src/test/java/edu/ucla/sspace/util/ObjectCounterTest.java

- Verified min()/max()

new file: src/test/java/edu/ucla/sspace/vector/AbstractDoubleVectorTest.java

- Added test for equals()