av9ash/GitBugs huggingface dataset:
Bug: Add cache in MLlib BinaryClassificationMetrics
Status: Closed | Priority: Minor | Resolution: Won't Fix
Created: 28/Feb/22 09:03
Affects Version/s: 2.2.3

we run some example code use BinaryClassificationEvaluator in MLlib, found that ShuffledRDD[28] at BinaryClassificationMetrics.scala:155 and UnionRDD[36] BinaryClassificationMetrics.scala:90 were used more than once but not cached.

We use spark-2.2.3 and found the code in branch master is still without cache, so we hope to improve it.

The example code is as follow:
{code:java}
import com.microsoft.ml.spark.lightgbm.LightGBMRegressor
import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.sql.types.{DoubleType, IntegerType}
import org.apache.spark.sql.{DataFrame, SparkSession}
object LightGBMRegressorTest {  def main(args: Array[String]): Unit = {    val spark: SparkSession = SparkSession.builder()
      .appName("LightGBMRegressorTest")
      .master("local[*]")
      .getOrCreate()    val startTime = System.currentTimeMillis()    var originalData: DataFrame = spark.read.option("header", "true")
      .option("inferSchema", "true")
      .csv("data/hour.csv")    val labelCol = "workingday"
    val cateCols = Array("season", "yr", "mnth", "hr")
    val conCols: Array[String] = Array("temp", "atemp", "hum", "casual", "cnt")
    val vecCols = conCols ++ cateCols    import spark.implicits._
    vecCols.foreach(col => {
      originalData = originalData.withColumn(col, $"$col".cast(DoubleType))
    })
    originalData = originalData.withColumn(labelCol, $"$labelCol".cast(IntegerType))    val assembler = new VectorAssembler().setInputCols(vecCols).setOutputCol("features")    val classifier: LightGBMRegressor = new LightGBMRegressor().setNumIterations(100).setNumLeaves(31)
      .setBoostFromAverage(false).setFeatureFraction(1.0).setMaxDepth(-1).setMaxBin(255)
      .setLearningRate(0.1).setMinSumHessianInLeaf(0.001).setLambdaL1(0.0).setLambdaL2(0.0)
      .setBaggingFraction(0.5).setBaggingFreq(1).setBaggingSeed(1).setObjective("binary")
      .setLabelCol(labelCol).setCategoricalSlotNames(cateCols).setFeaturesCol("features")
      .setBoostingType("gbdt")    val pipeline: Pipeline = new Pipeline().setStages(Array(assembler, classifier))    val Array(tr, te) = originalData.randomSplit(Array(0.7, .03), 666)
    val model = pipeline.fit(tr)
    val modelDF = model.transform(te)
    val evaluator = new BinaryClassificationEvaluator().setLabelCol(labelCol).setRawPredictionCol("prediction")
    println(evaluator.evaluate(modelDF))    println(s"time: ${System.currentTimeMillis() - startTime}" )
    System.in.read()
  }
}{code}
Bug: StructType explode
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 20/Feb/22 18:10
Affects Version/s: 3.2.1

Currently explode function only supports Array datatypes and Map datatypes but not StructType. Supporting StructType helps spark user's to transform datasets to a flatten one and this feature would be helpful with dealing semi-structured and unstructured datasets.
the idea is to support StructType in the first place and also add `prefix` and `postfix` option to it
Bug: Spark documentation build README is stale
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 14/Feb/22 22:16
Affects Version/s: 3.2.1

I was following docs/README.md to build documentation and found out that it's not complete. I had to install additional packages that is not documented but available in the [CI/CD phase |https://github.com/apache/spark/blob/c8b34ab7340265f1f2bec2afa694c10f174b222c/.github/workflows/build_and_test.yml#L526]and few more to finish the build process.

I will file a PR to change README.md to include these packages and improve the guide
Bug: AttributeError: 'LogisticRegressionModel' object has no attribute '_call_java'
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Feb/22 12:44
Affects Version/s: 2.4.0

Trying to invoke {{\_\_repr\_\_}} on {{pyspark.mllib.classification.LogisticRegressionModel}} leads to {{AttributeError}}:

{code:python}
>>> type(model)
<class 'pyspark.mllib.classification.LogisticRegressionModel'>
>>> model
Traceback (most recent call last):
  File /path/to/python3.9/site-packages/IPython/core/formatters.py:698 in __call__
    return repr(obj)
  File /path/to/spark/python/pyspark/mllib/classification.py:281 in __repr__
    return self._call_java("toString")
AttributeError: 'LogisticRegressionModel' object has no attribute '_call_java'
{code}

This problem was introduced SPARK-14712, where the method was added, with the same implementation, for both {{ml}} and {{mllib}}.
Bug: [proposal] collect & update statistics automatically when spark SQL is running
Status: Open | Priority: Minor | Resolution: 
Created: 20/Feb/22 09:53
Affects Version/s: 2.4.0

As we all know, table & column statistics are very important to spark SQL optimizer, however we have to collect & update them using 
{code:java}
analyze table tableName compute statistics{code}
It's a little inconvenient, so why can't we {color:#ff0000}collect & update statistics automatically{color} when a spark stage runs and finishes?

For example, when a insert overwrite table statement finishes, we can update a corresponding table statistics using SQL metrics. And in following queries, spark sql optimizer can use these statistics.

As we all know, it's a common case that we run daily batches using Spark SQLs, so a same SQL can run every day, and the SQL and its corresponding tables data change slowly. That means we can use statistics updated on yesterday to optimize current SQLs, of course can also adjust the important configs, such as spark.sql.shuffle.partitions

So we'd better add a mechanism to store every stage's statistics somewhere, and use it in new SQLs. Not just collect statistics after a stage finishes.

Of course, we'd better {color:#ff0000}add a version number to statistics{color} in case of losing efficacy

 

https://docs.google.com/document/d/1L48Dovynboi_ARu-OqQNJCOQqeVUTutLu8fo-w_ZPPA/edit#
Bug: Improve WritableColumnVector to better support null struct
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 10/Feb/22 18:53
Affects Version/s: 3.3.0

Currently {{WritableColumnVector}} of struct type requires to allocate space in all child vectors for null elements. This is not very space efficient. In addition, this model doesn't work well with Parquet vectorized scan for struct (in SPARK-34863).
Bug: Use Java 17 in K8S integration tests when setting spark-tgz
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Feb/22 10:12
Affects Version/s: 3.3.0

When setting parameters `spark-tgz` during integration tests, the error that `resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17` cannot be found occurs. This is due to the default value of `spark.kubernetes.test.dockerFile` being `resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17`. When using the tgz, the working directory is `${spark.kubernetes.test.unpackSparkDir}`, and the relative path `resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17` is invalid.
Bug: Clean up deprecated api usage of Ivy
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Feb/22 03:45
Affects Version/s: 3.3.0

{code:java}
[WARNING] [Warn] /spark-source/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:1459: [deprecation @ org.apache.spark.deploy.SparkSubmitUtils.resolveMavenCoordinates | origin=org.apache.ivy.Ivy.retrieve | version=] method retrieve in class Ivy is deprecated {code}
Bug: Upgrade tink to 1.6.1
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Feb/22 10:58
Affects Version/s: 3.3.0

release note:[https://github.com/google/tink/releases/tag/v1.6.1]

This version optimizes the performance of `AesGcmJce.encrypt` method, this method is used by AuthEngine#challenge and AuthEngine#response
Bug: Implement Strict Mode to prevent QUERY the entire table  
Status: Open | Priority: Minor | Resolution: 
Created: 24/Feb/22 17:37
Affects Version/s: 3.2.1

We are using Spark Thrift Server as a service to run Spark SQL queries along with Hive metastore as the metadata service.

We would like to restrict users from querying the entire table and force them to use {{WHERE clause in the query based on partition column (i.e. SELECT * FROM TABLE WHERE partition_column=<column_value>) *and*  LIMIT the output of the query when ORDER}} {{BY}} is used.

This behaviour is similar to what hive exposes as configuration

{{??hive.strict.checks.no.partition.filter??}}

{{??hive.strict.checks.orderby.no.limit??}}

and is described here: [https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1812]

and

[https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1816]

 

This is a pretty common usecase / feature that we meet in other tools as well,  like in BigQuery for example: [https://cloud.google.com/bigquery/docs/querying-partitioned-tables#require_a_partition_filter_in_queries]  .

It would be nice to have this feature implemented in Spark when hive support is enabled in a spark session.
Bug: Print resolved versions of platform JARs used by Spark in Yarn client
Status: In Progress | Priority: Minor | Resolution: 
Created: 28/Feb/22 22:32
Affects Version/s: 3.3.0

It would be nice if we could print the resolved versions. I think a sensible approach would be to resolve the symlinks in yarn.Client before uploading. This would also make the localized file name include the version number, so we could see it easily on the Environment SHS tab.
Bug: csv parser exception when quote and escape are both double-quote and a value is just "," and column pruning enabled
Status: Open | Priority: Minor | Resolution: 
Created: 25/Feb/22 16:01
Affects Version/s: 3.1.2

Workaround: disable column pruning.

Example pyspark code (from Databricks):
{noformat}
import pyspark
print(pyspark.version.__version__)

# enable column pruning (reset default value)
spark.conf.set('spark.sql.csv.parser.columnPruning.enabled', 'true')

dbutils.fs.put(file='/tmp/example.csv', contents='''"col1","b4_comma","comma","col4"
"","",",","x"
''', overwrite=True)

df = spark.read.csv(
    path='/tmp/example.csv'
    ,inferSchema=True
    ,header=True
    ,escape='"'
    ,multiLine=True
    ,unescapedQuoteHandling='RAISE_ERROR'
    ,mode='FAILFAST'
    )
ex = None
try:
    df.select(df.col1,df.comma).take(1)
except Exception as e:
    ex = e
    
if ex:
    print('[pruning] Exception is raised if b4_comma is NOT selected')
    
df.select(df.b4_comma, df.comma).take(1)
print('[pruning] No exception if b4_comma is selected')

ex = None
try:
    df.count()
except Exception as e:
    ex = e
    
if ex:
    print('[pruning] Exception raised by count')

print('\ndisabling pruning\n')
    
    
# disable column pruning
spark.conf.set('spark.sql.csv.parser.columnPruning.enabled', 'false')
df.select(df.col1,df.comma).take(1)
print('[no prune] No exception if b4_comma is NOT selected') {noformat}
 

Output:
{noformat}
3.1.2
Wrote 47 bytes.
[pruning] Exception is raised if b4_comma is NOT selected
[pruning] No exception if b4_comma is selected
[pruning] Exception raised by count

disabling pruning

[no prune] No exception if b4_comma is NOT selected {noformat}
Bug: No need to filter windows when windowDuration is multiple of slideDuration
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Feb/22 08:35
Affects Version/s: 3.2.1

At present, the sliding window adopts the form of expand + filter, but in some cases, filter is not necessary.

Filter is required if the sliding window is irregular. When the window length is divided by the slide length the result is an integer (I believe this is also the case for most work scenarios in practice for sliding window), there is no need to filter, which can save calculation resources and improve performance.
Bug: Sort the SparkSubmit debug output 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Feb/22 14:02
Affects Version/s: 3.2.1

When '-verbose' is passed to SparkSubmit it prints some useful debug information: Main class, Argument and Spark config.

I find it a bit hard to find information there because the arguments/configs are shuffled. I suggest to sort those before printing them.

 
{code:java}
 Main class:
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication
Arguments:
--main-class
--primary-java-resource
local:///opt/spark/examples/jars/spark-examples_2.13-3.3.0-SNAPSHOT.jar
org.apache.spark.examples.SparkPi
Spark config:
(spark.app.name,spark-on-k8s-app)
(spark.app.submitTime,1645106476125)
(spark.driver.cores,1)
(spark.driver.extraJavaOptions,-Dio.netty.tryReflectionSetAccessible=true)
(spark.driver.memory,2048m)
(spark.dynamicAllocation.enabled,true)
(spark.dynamicAllocation.shuffleTracking.enabled,true)
(spark.executor.cores,2)
(spark.executor.extraJavaOptions,-Dio.netty.tryReflectionSetAccessible=true)
(spark.executor.instances,3)
(spark.executor.memory,2048m)
(spark.jars,local:///opt/spark/examples/jars/spark-examples_2.13-3.3.0-SNAPSHOT.jar)
(spark.kubernetes.allocation.batch.delay,1)
(spark.kubernetes.allocation.batch.size,3)
(spark.kubernetes.authenticate.driver.serviceAccountName,spark-account-name)
(spark.kubernetes.driver.container.image,spark/spark:3.3.0-SNAPSHOT-scala_2.13-11-jre-slim)
(spark.kubernetes.executor.container.image,spark/spark:3.3.0-SNAPSHOT-scala_2.13-11-jre-slim)
(spark.kubernetes.namespace,spark-on-k8s)
(spark.master,k8s://https://192.168.49.2:8443)
(spark.network.timeout,300)
(spark.submit.deployMode,cluster)
(spark.submit.pyFiles,)
Classpath elements:{code}
 

The "Parsed arguments:" order is hardcoded at org.apache.spark.deploy.SparkSubmitArguments#toString, so they are still shuffled.
Bug: Should't check temp/external/ifNotExists with visitReplaceTable when parser
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Feb/22 09:45
Affects Version/s: 3.2.0

Spark does not support replace table syntax such as CREATE OR REPLACE TEMPORARY TABLE.../REPLACE EXTERNAL TABLE/REPLACE ... IF NOT EXISTS, And we don't need to check these tokens
Bug: Upgarde junit4 to 4.13.2 and upgrade corresponding junit-interface to 0.13.3
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Feb/22 10:54
Affects Version/s: 3.3.0

Upgrade junit4 to 4.13.2 and upgrade corresponding junit-interface to 0.13.3 refer to https://github.com/sbt/junit-interface/
Bug: Desc column stats (min, max) for timestamp type is not consistent with the value due to time zone difference
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Feb/22 12:48
Affects Version/s: 3.1.2

Currently timestamp column's stats (min/max) are stored in UTC in metastore, and when desc its min/max column stats, they are also shown in UTC.

As a result, for users not in UTC, the column stats (shown to users) are not consistent with the actual value, which causes confusion.

For example:

{noformat}
spark-sql> create table tab_ts_master (ts timestamp) using parquet;

spark-sql> insert into tab_ts_master values make_timestamp(2022, 1, 1, 0, 0, 1.123456), make_timestamp(2022, 1, 3, 0, 0, 2.987654);

spark-sql> select * from tab_ts_master;
2022-01-01 00:00:01.123456
2022-01-03 00:00:02.987654

spark-sql> set spark.sql.session.timeZone;
spark.sql.session.timeZone	Asia/Shanghai

spark-sql> analyze table tab_ts_master compute statistics for all columns;

spark-sql> desc formatted tab_ts_master ts;
col_name	ts
data_type	timestamp
comment	NULL
min	2021-12-31 16:00:01.123456
max	2022-01-02 16:00:02.987654
num_nulls	0
distinct_count	2
avg_col_len	8
max_col_len	8
histogram	NULL
{noformat}
Bug: Unify the output of df.explain and "explain " if plan is command
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 18/Feb/22 03:23
Affects Version/s: 3.2.0

This pr aims to unify the out of df.explain and "explain" sql if plan is a command
 * unify the out of df.explain and "explain" sql if plan is a command
 * Make the output of explain unambiguous if plan is a command

 

Lets say have a query like "show tables", we want to explain it

Before this pr:
{code:java}
== Parsed Logical Plan ==
'ShowTables [namespace#62, tableName#63, isTemporary#64]
+- 'UnresolvedNamespace== Analyzed Logical Plan ==
namespace: string, tableName: string, isTemporary: boolean
ShowTablesCommand default, [namespace#62, tableName#63, isTemporary#64], false== Optimized Logical Plan ==
CommandResult [namespace#62, tableName#63, isTemporary#64], Execute ShowTablesCommand, [[default,lt,false], [default,people,false], [default,person,false], [default,rt,false], [default,t,false], [default,t1,false], [default,t10086,false], [default,txtsource,false]]
   +- ShowTablesCommand default, [namespace#62, tableName#63, isTemporary#64], false== Physical Plan ==
CommandResult [namespace#62, tableName#63, isTemporary#64]
   +- Execute ShowTablesCommand
         +- ShowTablesCommand default, [namespace#62, tableName#63, isTemporary#64], false {code}
After this pr:
{code:java}
== Parsed Logical Plan ==
'ShowTables [namespace#0, tableName#1, isTemporary#2]
+- 'UnresolvedNamespace== Analyzed Logical Plan ==
namespace: string, tableName: string, isTemporary: boolean
ShowTables [namespace#0, tableName#1, isTemporary#2]
+- ResolvedNamespace V2SessionCatalog(spark_catalog), [default]== Optimized Logical Plan ==
ShowTables [namespace#0, tableName#1, isTemporary#2]
+- ResolvedNamespace V2SessionCatalog(spark_catalog), [default]== Physical Plan ==
ShowTables [namespace#0, tableName#1, isTemporary#2], V2SessionCatalog(spark_catalog), [default] {code}
Bug: Clean up unused parameters in private methods signature
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Feb/22 12:31
Affects Version/s: 3.3.0

The private method `castDecimalToIntegralTypeCode` in Cast.scala as follows:

 
{code:java}
private[this] def castDecimalToIntegralTypeCode(
      ctx: CodegenContext,
      integralType: String,
      catalogType: String): CastFunction = {
    if (ansiEnabled) {
      (c, evPrim, evNull) => code"$evPrim = $c.roundTo${integralType.capitalize}();"
    } else {
      (c, evPrim, evNull) => code"$evPrim = $c.to${integralType.capitalize}();"
    }
  } {code}
`ctx` and `catalogType` are useless
Bug: When creating a Hive table, fail early if all the columns are partitioned columns
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Feb/22 09:39
Affects Version/s: 3.2.0

In Hive the schema and partition columns must be disjoint sets, if hive table which all columns are partitioned columns, so that other columns is empty, it will fail when Hive create table, error msg as follow:

`

throw new HiveException(
"at least one column must be specified for the table")

`

So when creating a Hive table, fail early if all the columns are partitioned columns,
Bug: Delete the unused `dataType` specified in the definition of `IntervalColumnAccessor`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Feb/22 07:51
Affects Version/s: 3.3.0

SPARK-30066 introduce `IntervalColumnAccessor` and it  accepts 2 constructor parameters: `buffer` and `dataType`, the `dataType` is unused because the parameter passed to `BasicColumnAccessor` is a constant `CALENDAR_INTERVAL`
Bug: Fix `QueryExecution.debug#toFile` use the passed in `maxFields` when `explainMode` is `CodegenMode`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Feb/22 07:24
Affects Version/s: 3.3.0

`QueryExecution.debug#toFile` method supports passing in `maxFields` and this parameter will be passed down when `explainMode` is SimpleMode, ExtendedMode, or CostMode, but the passed down `maxFields` was ignored because `QueryExecution#stringWithStats` overrides it with `SQLConf.get.maxToStringFields` at present
Bug: Use try-with-resources in Level/RocksDBSuite.java
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Feb/22 13:56
Affects Version/s: 3.3.0

Manually compiled rocksdbjni with rocksdb 6.29.fb on Apple Silicon/M1, then test `org.apache.spark.util.kvstore.RocksDBSuite` using the generated jar, there is a VM crash caused by resource leak
Bug: Change of unionByName parameter
Status: Open | Priority: Minor | Resolution: 
Created: 12/Feb/22 23:07
Affects Version/s: 3.3.0

Hello,

I had a quick question about the unionByName function. This function currently seems to accept a parameter- "allowMissingColumns"- that allows some tolerance to merging datasets with different schemas [here|[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L2170]]; but the implementation is currently a bit restrictive, i.e., with the second parameter being a boolean, it is only possible to make unionByName add all columns from both dataframes at the moment. We have other use cases in our workflows- for example, to take only column names that are in both dataframes (and I'm assuming that other users will have different merge strategies in mind also). Does it seem reasonable to extend the parameter from "allowMissingColumns" to a "mode" string-type parameter natively in Spark? If so, I'm happy to make a PR to achieve this (the change would involve amending the [ResolveUnion.scala|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala] utility to make it more flexible in merging columns; to a user it would look a lot more like the 'join' operator, where a join strategy is selected). 

I've posted this question on the dev mailing list also; happy to continue the conversation there if that is preferable.
Bug: Unified use `DataType.catalogString` as `targetType` of `QueryExecutionErrors#castingCauseOverflowError`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Feb/22 05:39
Affects Version/s: 3.3.0

SPARK-33541 introduces `QueryExecutionErrors#castingCauseOverflowError` and 

there are 2 ways to input parameter `targetType` in spark:
 * Use `DataType.catalogString` as `targetType`, such as use in `Cast.scala` and `IntervalUtils.scala`
 * Use custom literal such as `short`,`int` and `long` in `Decimal.scala` and `numberics.scala`

 

And we can use `DataType.catalogString` uniformly
Bug: Use OffHeap memory if configured in vectorized DeltaByteArray reader
Status: Open | Priority: Minor | Resolution: 
Created: 09/Feb/22 18:01
Affects Version/s: 3.2.1

The VectorizedDeltaByteArray reader allocates some vectors for internal use that are allocated off of heap memory. Depending on the configuration, the vectors should be either on-heap or off-heap. 
To support off-heap vectors, the vectorized readers must be made closable and memory allocated for the vectors must be freed on close.
Bug: Shuffle by rand could lead to incorrect answers when ShuffleFetchFailed happend
Status: In Progress | Priority: Minor | Resolution: 
Created: 09/Feb/22 08:48
Affects Version/s: 3.3.0

When we do shuffle on indeterminate expressions such as rand, and ShuffleFetchFailed happend, we may get incorrent result since it only retries failed map tasks.

We try to fix this by retry all upstream map tasks in this situation.
Bug: Make ArrowColumnVector Extendable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Feb/22 23:39
Affects Version/s: 3.3.0

Some Spark extension libraries need to extend ArrowColumnVector.java. For now, it is impossible as ArrowColumnVector class is final and the accessors are all private.

For example, Rapids copies the entire ArrowColumnVector class in order to work around the issue

[https://github.com/NVIDIA/spark-rapids/blob/main/sql-plugin/src/main/java/org/apache/spark/sql/vectorized/rapids/AccessibleArrowColumnVector.java]

Proposing to relax private/final restrictions to make ArrowColumnVector extendable.