av9ash/GitBugs huggingface dataset:
Bug: Use fine-grained lock in SessionCatalog.tableExists
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Mar/21 03:24
Affects Version/s: 3.1.1

We have modified the underlying hive meta store which a different hive database is placed in its own shard for performance. However, we found that  the synchronized limits the concurrency, we would like to fix it.

Related jstack trace like following:
{code:java}
"http-nio-7070-exec-257" #19961734 daemon prio=5 os_prio=0 tid=0x00007f45f4ce1000 nid=0x1a85e6 waiting for monitor entry [0x00007f45949df000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.tableExists(SessionCatalog.scala:415)
    - waiting to lock <0x000000011d983d90> (a org.apache.spark.sql.catalyst.catalog.SessionCatalog)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireTableExists(SessionCatalog.scala:185)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableMetadata(SessionCatalog.scala:430)
    at org.apache.spark.sql.DdlOperation$.getTableDesc(SourceFile:123)
    at org.apache.spark.sql.DdlOperation.getTableDesc(SourceFile)
...
{code}
we fixed as discussed in mail [http://mail-archives.apache.org/mod_mbox/spark-dev/202103.mbox/browser]
Bug: SessionCatalog's createTempView/createGlobalTempView should accept TemporaryViewRelation 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/21 19:15
Affects Version/s: 3.2.0

SessionCatalog's createTempView/createGlobalTempView currently accept LogicalPlan to store temp views, but once all the commands are migrated to store `TemporaryViewRelation`, it should accept TemporaryViewRelation instead.

Once this is done, ViewHelper.needsToUncache can remove the following safely: case p => !p.sameResult(aliasedPlan)
Bug: Move simplify expression rules before push down rules
Status: In Progress | Priority: Major | Resolution: 
Created: 19/Mar/21 15:51
Affects Version/s: 3.2.0

For example:
{code:scala}
spark.sql("CREATE TABLE t1 using parquet AS SELECT id AS a FROM range(10)")
spark.sql("CREATE TABLE t2 using parquet AS SELECT id AS b FROM range(10)")
spark.sql("SELECT * FROM t1 INNER JOIN t2 ON (a = b AND true)").explain
{code}
{noformat}
=== Applying Rule org.apache.spark.sql.catalyst.optimizer.PushDownPredicates ===
!Join Inner, ((a#4L = b#5L) AND true)   Join Inner, (a#4L = b#5L)
!:- Relation default.t1[a#4L] parquet   :- Filter true
!+- Relation default.t2[b#5L] parquet   :  +- Relation default.t1[a#4L] parquet
!                                       +- Relation default.t2[b#5L] parquet
{noformat}


BooleanSimplification should before PushDownPredicates.
Bug: Upgrade Jackson to 2.12.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/21 04:58
Affects Version/s: 3.2.0

Upgrade Jackson to 2.12.2 to make it easy to upgrade Avro 1.10.2.

More details:
 [https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.12]
Bug: Fix incorrect join condition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Mar/21 02:31
Affects Version/s: 3.2.0

[https://github.com/apache/spark/blob/5988d2846450f8ec72fefcd2067f12819463cc4b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala#L150]

[https://github.com/apache/spark/blob/04e53d2e3c9dc993b8ba88c5ef0bcf0a8a9b06b2/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeLimitZeroSuite.scala#L77]

[https://github.com/apache/spark/blob/04e53d2e3c9dc993b8ba88c5ef0bcf0a8a9b06b2/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeLimitZeroSuite.scala#L89]

[https://github.com/apache/spark/blob/04e53d2e3c9dc993b8ba88c5ef0bcf0a8a9b06b2/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeLimitZeroSuite.scala#L91]
Bug: sql method in UnresolvedAttribute, AttributeReference and Alias don't quote qualified names properly.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/21 07:21
Affects Version/s: 3.0.2

UnresolvedReference, AttributeReference and Alias which take qualifier don't quote qualified names properly.

One instance is reported in SPARK-34626.
Other instances are like as follows.

{code}
UnresolvedAttribute("a`b"::"c.d"::Nil).sql
a`b.`c.d` // expected: `a``b`.`c.d`

AttributeReference("a.b", IntegerType)(qualifier = "c.d"::Nil).sql
c.d.`a.b` // expected: `c.d`.`a.b`

Alias(AttributeReference("a", IntegerType)(), "b.c")(qualifier = "d.e"::Nil).sql
`a` AS d.e.`b.c` // expected: `a` AS `d.e`.`b.c`
{code}
Bug: Remove all SQLConf.get if extends from SQLConfHelper
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/21 18:29
Affects Version/s: 3.2.0

We do not need SQLConf.get if extends from SQLConfHelper. For example: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala#L89]. We can use {{conf.exchangeReuseEnabled}}.
Bug: ANSI mode: Abs throws exception if input is out of range
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/21 08:05
Affects Version/s: 3.2.0

For the following cases, ABS should throw exceptions since the result is out of the range of result data type in ANSI mode.

{code:java}
SELECT abs(${Int.MinValue});
SELECT abs(${Long.MinValue});
{code}
Bug: Add tableType for SHOW TABLES to distinguish view and tables
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Mar/21 03:05
Affects Version/s: 3.0.2

SHOW TABLES has an output column `isTemporary` which only indicates if a view is a local temp view that is not enough for users if they want pipeline Spark commands, such as SHOW TABLES -> DROP TABLE/VIEW
Bug: CVEs in library dependencies
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 04/Mar/21 10:57
Affects Version/s: 3.1.1

There are several CVEs in dependency libraries in the current version of pyspark, including shaded ones under JARs such as 
 * htrace-core4-4.1.0-incubating.jar:jackson-databind

Are these already being looked at? If not, could they be updated to reference newer versions? Thanks.
|CVE ID|Type|Severity|Packages|Package Version|CVSS|Fix Status|
|CVE-2017-18640|java|high|org.yaml_snakeyaml|1.24|7.5|fixed in 1.26|
|CVE-2020-25649|java|high|com.fasterxml.jackson.core_jackson-databind|2.10.0|7.5|fixed in 2.10.5.1, 2.9.10.7, 2.6.7.4|
|CVE-2020-35491|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2020-35490|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2018-14718|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.7|
|CVE-2018-7489|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.5, 2.8.11.1, 2.7.9.3|
|CVE-2020-25649|java|high|com.fasterxml.jackson.core_jackson-databind|2.10.0|7.5|fixed in 2.10.5.1, 2.9.10.7, 2.6.7.4|
|CVE-2020-35491|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2020-35490|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2018-14718|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.7|
|CVE-2018-7489|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.5, 2.8.11.1, 2.7.9.3|
|CVE-2017-18640|java|high|org.yaml_snakeyaml|1.24|7.5|fixed in 1.26|
|CVE-2020-25649|java|high|com.fasterxml.jackson.core_jackson-databind|2.10.0|7.5|fixed in 2.10.5.1, 2.9.10.7, 2.6.7.4|
|CVE-2020-35491|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2020-35490|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2018-14718|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.7|
|CVE-2018-7489|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.5, 2.8.11.1, 2.7.9.3|
|CVE-2017-18640|java|high|org.yaml_snakeyaml|1.24|7.5|fixed in 1.26|
|CVE-2019-17195|java|critical|com.nimbusds_nimbus-jose-jwt|4.41.1|9.8|fixed in 7.9|

 

Similar issues, for reference:
 # CAMEL-14640
 # HADOOP-16690
 # ZEPPELIN-4657
Bug: ARM CI failed due to download hive-exec failed
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 05/Mar/21 14:58
Affects Version/s: 3.1.2

The download failed (org.apache.hive#hive-exec;3.0.0!hive-exec.jar) was happened in recent spark-master-test-maven-arm build tests.

But it's not reproduced and all hive test passed in my arm and x86 local env.

and the jenkins x86 test like [2] is also unstable due to other reason, so looks like we can't find a valid reference.

[1] [https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/]

[2] [https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/]
Bug: UnresolvedAttribute.sql may return an incorrect sql
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/21 23:25
Affects Version/s: 3.0.2

UnresolvedAttribute("a" :: "b" :: Nil).sql

returns
{code:java}
`a.b`{code}
The correct one should be either a.b or
{code:java}
`a`.`b`{code}
Bug: Overflow in round trip conversion from micros to duration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/21 17:21
Affects Version/s: 3.2.0

The code below fails with long overflow:
{code:scala}
scala> import org.apache.spark.sql.catalyst.util.IntervalUtils._
import org.apache.spark.sql.catalyst.util.IntervalUtils._

scala> val minDuration = microsToDuration(Long.MinValue)
minDuration: java.time.Duration = PT-2562047788H-54.775808S

scala> durationToMicros(minDuration)
java.lang.ArithmeticException: long overflow
  at java.lang.Math.multiplyExact(Math.java:892)
  at org.apache.spark.sql.catalyst.util.IntervalUtils$.durationToMicros(IntervalUtils.scala:782)
  ... 49 elided
{code}
Bug: Py4J 0.10.9.1 is out with bug fixes. we should better upgrade in PySpark as well.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Mar/21 09:42
Affects Version/s: 3.2.0

Py4J 0.10.9.2 is out with bug fixes. we should better upgrade in PySpark as well.
Bug: org.apache.spark.sql.DataFrameNaFunctions.replace() fails for column name having a dot
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/21 10:09
Affects Version/s: 3.1.0

Code to reproduce the issue:
{code:java}
import org.apache.spark.sql.SparkSession

object ColumnNameWithDot {

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("Simple Application")
      .config("spark.master", "local").getOrCreate()

    spark.sparkContext.setLogLevel("OFF")

    import spark.implicits._
    val df = Seq(("abc", 23), ("def", 44), ("n/a", 0)).toDF("ColWith.Dot", "Col.2")

    df.na.replace("`ColWith.Dot`", Map("n/a" -> "Unknown"))
      .show()
  }
}
{code}
Bug: Revise the type coercion section of ANSI Compliance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/21 14:25
Affects Version/s: 3.2.0

1. Fix the table of valid type coercion combinations. Binary type should be allowed casting to String type and disallowed casting to Numeric types.
2. Summary all the `CAST`s that can cause runtime exceptions.
Bug: OrcColumnarBatchReader uncaught exception
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 02/Mar/21 09:54
Affects Version/s: 2.4.5

RecordReaderInterator#hasNext doesn't handle exception thrown from OrcColumnarBatchReader#nextKeyValue
{code:java}
// code placeholder
override def hasNext: Boolean = {
  if (!finished && !havePair) {
    finished = !rowReader.nextKeyValue // Exception thrown from here
    if (finished) {
      // Close and release the reader here; close() will also be called when the task
      // completes, but for tasks that read from many files, it helps to release the
      // resources early.
      close()
    }
    havePair = !finished
  }
  !finished
}

{code}
[https://github.com/apache/spark/blob/96f51378b598501cc5dc5791f794c8586a48b355/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/RecordReaderIterator.scala#L37]

 

!report.PNG!
Bug: Enable Arrow optimization for additional supported types when using SparkR
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Mar/21 22:54
Affects Version/s: 3.0.1

I am opening this issue to track pull request here: [https://github.com/apache/spark/pull/31744]

 

The R {{arrow}} package now supports {{FloatType}}, {{BinaryType}}, {{ArrayType}},{{StructType}}. This was brought to my attention by Neal Richardson, maintainer of the R Arrow package, in the comments on this issue: https://issues.apache.org/jira/browse/ARROW-3783

The proposed changes allows SparkR users to leverage arrow optimization for the additional types supported. Documentation for currently supported types can be viewed at [https://arrow.apache.org/docs/r/articles/arrow.html#arrow-to-r]
Bug: Remove GlobalLimit operator if it's child max row <= limit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/21 02:11
Affects Version/s: 3.2.0


{code:scala}
  test("Remove GlobalLimit operator if it's child max row <= limit") {
    val query = GlobalLimit(100, testRelation)
    val optimized = Optimize.execute(query.analyze)
    comparePlans(optimized, testRelation)
  }
{code}
Bug: DPP support RLIKE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/21 10:01
Affects Version/s: 3.2.0

Support this case:
{code:sql}
SELECT date_id, product_id FROM fact_sk f
JOIN dim_store s
ON f.store_id = s.store_id WHERE s.country RLIKE '[DE|US]'
{code}
Bug: Do not delete shuffle file on executor lost event when using remote shuffle service
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 03/Mar/21 00:04
Affects Version/s: 3.2.0

There are multiple work going on with disaggregated/remote shuffle service (e.g. [LinkedIn shuffle|https://engineering.linkedin.com/blog/2020/introducing-magnet], [Facebook shuffle service|https://databricks.com/session/cosco-an-efficient-facebook-scale-shuffle-service], [Uber shuffle service|https://github.com/uber/RemoteShuffleService]). Such remote shuffle service is not Spark External Shuffle Service. It could be third party shuffle solution and user uses it by setting spark.shuffle.manager. In those systems, shuffle data will be stored on different server other than executor. Spark should not mark shuffle data lost when the executor is lost. We could add a Spark configuration to control this behavior. By default, Spark still mark shuffle file lost. For disaggregated/remote shuffle service, people could set the configure to not mark shuffle file lost.
Bug: Consider supporting distribution and ordering definitions during table creation 
Status: Open | Priority: Major | Resolution: 
Created: 02/Mar/21 04:40
Affects Version/s: 3.2.0

I think it makes sense to allow users to control the distribution and ordering of data during table creation. Spark should parse such expressions, map them into {{Distribution}} and {{SortOrder}} and pass to data sources.

The SQL syntax requires further consideration.
Bug: SessionCatalog#requireDbExists is called too many times affects SQL performance
Status: Open | Priority: Minor | Resolution: 
Created: 04/Mar/21 08:48
Affects Version/s: 2.4.5

Almost all SQL need to check if database exist before execution.

Usually database is stable. Every check needs to connect to Metastore and check if the DB exists in Databse. This behavior could be too expensive.

 

A proposal is to add a switch to control if it's necessary to check DB's existance.

 

[requireDbExists]([https://github.com/apache/spark/blob/53e4dba7c489ac5c0ad61f0121c4e247de5b485c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala#L197)]

 

!db_existance.PNG!
Bug: inferTimestamp option is missing from the list of options in DataFrameReader.json.
Status: Open | Priority: Minor | Resolution: 
Created: 09/Mar/21 19:40
Affects Version/s: 3.0.1

{color:#e01e5a}inferTimestamp{color} option is missing in the list of options in {color:#e01e5a}DataFrameReader.json{color} method in the API docs missing from the [Scaladocs here|[DataFrameReader.json|https://github.com/apache/spark/blob/v3.1.1/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L432-L520]].

Simiarly in the [Pyspark docs|[pyspark.sql.DataFrameReader.json|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrameReader.json.html?highlight=json#pyspark.sql.DataFrameReader.json]] as well.

However we have this blurb in the [migration guide|[Spark 3.0 to 3.0.1 migration guide|https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-30-to-301]]
 * In Spark 3.0, JSON datasource and JSON function {{schema_of_json}} infer TimestampType from string values if they match to the pattern defined by the JSON option {{timestampFormat}}. Since version 3.0.1, the timestamp type inference is disabled by default. Set the JSON option {{inferTimestamp}} to {{true}} to enable such type inference.

We should add this in the documentation as well as there is a possibility that the Data Engineers might not be aware of this option.
Bug: Add Spark AM Log link in case of master as yarn and deploy mode as client
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Mar/21 14:11
Affects Version/s: 3.1.1

On Running Spark job with yarn and deployment mode as client, Spark Driver and Spark Application master launch in two separate containers. In various scenarios there is need to see Spark Application master logs to see the resource allocation, Decommissioning status and other information shared between yarn RM and Spark Application master.

Till now the only way to check this by finding the container id of the AM and check the logs either using Yarn utility or Yarn RM Application History server. 

This Jira is for adding the spark AM log link for spark job running in the client mode for yarn. Instead of searching the container id and then find the logs. We can directly check in the Spark UI
Bug: Add code-gen for all join types of sort merge join
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Mar/21 02:24
Affects Version/s: 3.2.0

Currently sort merge join only supports inner join type ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala#L374] ). We added code-gen for other join types internally in our fork and saw obvious CPU performance improvement. Create this Jira to propose to merge back to upstream.
Bug: Replaces `OriginalType` with `LogicalTypeAnnotation` in VectorizedColumnReader
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Mar/21 08:38
Affects Version/s: 3.2.0

{{OriginalType}} and {{DecimalMetadata}} has been marked as {{@Deprecated}} in new Parquet code.

{{Apache Parquet}} suggest us replace {{OriginalType}} with {{LogicalTypeAnnotation}} and replace {{DecimalMetadata}} with {{DecimalLogicalTypeAnnotation.}}

{{}}

The files to be changed are as follows:
 * VectorizedColumnReader.java
 * ParquetFilters.scala
 * ParquetReadSupport.scala
 * ParquetRowConverter.scala
 * ParquetSchemaConverter.scala

 

{{}}
Bug: Support null in conversions to and from Arrow
Status: Open | Priority: Minor | Resolution: 
Created: 10/Mar/21 15:58
Affects Version/s: 3.1.1

Looks like a regression from or related to SPARK-33489.

I got the following running spark 3.1.1. with koalas 1.7.0
{code:java}
TypeError                                 Traceback (most recent call last)
/var/scratch/miniconda3/lib/python3.8/site-packages/pyspark/sql/udf.py in returnType(self)
    100             try:
--> 101                 to_arrow_type(self._returnType_placeholder)
    102             except TypeError:

/var/scratch/miniconda3/lib/python3.8/site-packages/pyspark/sql/pandas/types.py in to_arrow_type(dt)
     75     else:
---> 76         raise TypeError("Unsupported type in conversion to Arrow: " + str(dt))
     77     return arrow_type

TypeError: Unsupported type in conversion to Arrow: NullType
{code}
Bug: can class pyspark.mllib.evaluation.RegressionMetrics() accept DataFrame as the parameter?
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 15/Mar/21 03:19
Affects Version/s: 2.4.6

class pyspark.mllib.evaluation.RegressionMetrics(predictionAndObservations) can only accept  an RDD of (prediction, observation) pairs as the parameter.

Can it also accept the DataFrame as the parameter? Which would be much more convenient.
Bug: Improve Spark SQL Source Filter to allow pushdown of filters span multiple columns
Status: Open | Priority: Minor | Resolution: 
Created: 10/Mar/21 16:38
Affects Version/s: 3.0.0

The current org.apache.spark.sql.sources.Filter abstract class only allows pushdown of filters on single column or sum of products of multiple such single-column filters.

Filters on multiple columns cannot be pushed down through this Filter subclass to source, e.g. from TPC-H benchmark on lineitem table:

(l_commitdate#11 < l_receiptdate#12)

(l_shipdate#10 < l_commitdate#11)

 

The current design probably originates from the point that columnar source has a hard time supporting these cross-column filters. But with batching implemented in columnar sources, they can still support cross-column filters.  This issue tries to open up discussion on a more general Filter interface to allow pushing down cross-column filters.
Bug: Kryo serializer not working with Spark SQL module
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 08/Mar/21 02:36
Affects Version/s: 3.0.1

Hey all,

I apologize if this has already been explored, but I keep getting this error whenever I have the Kryo registration required in my Spark app. The error is below; I have tried many a different types of combination to register this class but I just can't get anything to work, and so any help will be highly appreciated!

 
{code:java}
Caused by: com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.execution.datasources.InMemoryFileIndex$Seriali
zableBlockLocation[]
Note: To register this class use: kryo.register(org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[].class);
Serialization trace:
blockLocations (org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableFileStatus)
array (scala.collection.mutable.ArrayBuffer)

{code}
 

I've tried all kinds of things to get these classes to be registered, but it's just not working for me..