av9ash/GitBugs huggingface dataset:
Bug: Add the option to use physical statistics to avoid large tables being broadcast
Status: Closed | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 04:33
Affects Version/s: 3.2.0

Currently when AQE's queryStage is not materialized, it uses the stats of the logical plan to estimate whether the plan can be converted to BHJ, and in some scenarios the estimated value is several orders of magnitude smaller than the actual broadcast data, which can lead to large tables being broadcast
Bug: Bad error message when using non-existing named window
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 08:44
Affects Version/s: 3.3.0

{code:java}
CREATE TABLE employees 
(name STRING, dept STRING, salary INT, age INT);

SELECT AVG(age) OVER win AS salary, 
AVG(salary) OVER win AS avgsalary,
MIN(salary) OVER win AS minsalary,
MAX(salary) OVER win AS maxsalary,
COUNT(1) OVER win AS numEmps
FROM employees;

Error in query: unresolved operator 'Aggregate [unresolvedwindowexpression(avg(age#43), WindowSpecReference(win)) AS salary#34, unresolvedwindowexpression(avg(salary#42), WindowSpecReference(win)) AS avgsalary#35, unresolvedwindowexpression(min(salary#42), WindowSpecReference(win)) AS minsalary#36, unresolvedwindowexpression(max(salary#42), WindowSpecReference(win)) AS maxsalary#37, unresolvedwindowexpression(count(1), WindowSpecReference(win)) AS numEmps#38];
'Aggregate [unresolvedwindowexpression(avg(age#43), WindowSpecReference(win)) AS salary#34, unresolvedwindowexpression(avg(salary#42), WindowSpecReference(win)) AS avgsalary#35, unresolvedwindowexpression(min(salary#42), WindowSpecReference(win)) AS minsalary#36, unresolvedwindowexpression(max(salary#42), WindowSpecReference(win)) AS maxsalary#37, unresolvedwindowexpression(count(1), WindowSpecReference(win)) AS numEmps#38]
+- SubqueryAlias spark_catalog.default.employees
+- HiveTableRelation [`default`.`employees`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [name#40, dept#41, salary#42, age#43], Partition Cols: []]
{code}
Bug: Support using SQL string as a general Data Source V2 Expression
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/21 15:55
Affects Version/s: 3.3.0

In Data Source V2 APIs, we kind of need to fork the catalyst Expressions and create their public versions. This is going to be a long journey. In the meanwhile, we can add a general Data Source V2 expression that contains standard SQL string (following ANSI standard).
Bug: Upgrade aircompressor to 1.21
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 03:19
Affects Version/s: 3.2.0

We need to upgrade `aircompressor` to bring the latest bug fix. Aircompressor 1.17/1.18/1.19/1.20 has this bug.
- https://github.com/airlift/aircompressor/commit/1e364f713390008eada1daa451e7b42cd6647250
Bug: Support access and read parquet file by column index
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Sep/21 08:16
Affects Version/s: 3.3.0

Add a config spark.sql.parquet.columnIndexAccess 

When true, we access the parquet files by column index instead of catalyst schema mapping at the executor side. This is useful when the parquet file meta is inconsistent with those in Metastore
Bug: Upgrade commons-io to 2.11.0 and revert change of SPARK-36456
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Sep/21 08:02
Affects Version/s: 3.3.0

{{`JavaUtils.closeQuietly`}} looks to be slightly different from `{{IOUtils.closeQuietly`}} in default behavior. Swallowing IOException is same, but the former logs it as ERROR while the latter doesn't log by default. 

 

And Apache commons-io community decided to retain the `IOUtils.closeQuietly` method in new version and removed deprecated annotation.

 

So we can upgrade `{{commons-io`}} to version 2.11.0, and revert the change of SPARK-36456

to maintain original behavior(don't print error log).
Bug: bugs in MIniLSH
Status: Open | Priority: Minor | Resolution: 
Created: 10/Sep/21 08:37
Affects Version/s: 2.1.1

This is about MinHashLSH algorithm.
To get the similartiy dataframe DFA and DFB, I  used MinHashLSH  approxSimilarityJoin function.  But there are some missing data in the result.
the example in documents is no problem  [https://spark.apache.org/docs/latest/ml-features.html#minhash-for-jaccard-distance|http://example.com] 
but when the data based on distributed system(hive, more than one node)
there will be some missing data. 
for example    vectora= vectorb. but it no in the reslut of  approxSimilarityJoin, even though 

"threshold"  more than 1 .

I think  maybe the problem is  in these codes
{code:java}
// part1
override protected[ml] def createRawLSHModel(inputDim: Int): MinHashLSHModel1 = {
  require(inputDim <= MinHashLSH.HASH_PRIME,
    s"The input vector dimension $inputDim exceeds the threshold ${MinHashLSH.HASH_PRIME}.")
  val rand = new Random($(seed)) 
  val randCoefs: Array[(Int, Int)] = Array.fill($(numHashTables)) {
    (1 + rand.nextInt(MinHashLSH.HASH_PRIME - 1), rand.nextInt(MinHashLSH.HASH_PRIME - 1))
  }
  new MinHashLSHModel1(uid, randCoefs)
}

// part2
@Since("2.1.0")
override protected[ml] val hashFunction: Vector => Array[Vector] = {
  elems: Vector => {
    require(elems.numNonzeros > 0, "Must have at least 1 non zero entry.")
    val elemsList = elems.toSparse.indices.toList
    val hashValues = randCoefficients.map { case (a, b) =>
      elemsList.map { elem: Int =>
        ((1 + elem) * a + b) % MinHashLSH.HASH_PRIME
      }.min.toDouble
    }
    // TODO: Output vectors of dimension numHashFunctions in SPARK-18450
    hashValues.map(Vectors.dense(_))
  }
{code}

 val r1 = new scala.util.Random(1)

r1.nextInt(1000)  // -> 985

val r2 = new scala.util.Random(2)

r2.nextInt(1000)  // -> 108 - 
val r3 = new scala.util.Random(1)

r3.nextInt(1000)  // -> 985 - because seeded just as `r1`
r3.nextInt(1000).  //-> 588
{{}}the reason maybe is above.  if  random is only  initialized once .  random.nextInt() will get different result every time ,like r3. r3.nextInt(1000) // -> 985   r3.nextInt(1000).  //-> 588

so the code 
val rand = new Random($(seed)) in  def createRawLSHModel  move to hashFunction is better
. every worker will initialize random class. and every worker get same data
Bug: "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Sep/21 05:50
Affects Version/s: 3.3.0

Execute

 
{code:java}
build/mvn clean install  -pl core -am -Dtest=none -DwildcardSuites=org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite
{code}
with JDK 17,
{code:java}
ChunkedByteBufferOutputStreamSuite:
- empty output
- write a single byte
- write a single near boundary
- write a single at boundary
- single chunk output
- single chunk output at boundary size
- multiple chunk output
- multiple chunk output at boundary size
*** RUN ABORTED ***
  java.lang.OutOfMemoryError: Java heap space
  at java.base/java.lang.Integer.valueOf(Integer.java:1081)
  at scala.runtime.BoxesRunTime.boxToInteger(BoxesRunTime.java:67)
  at org.apache.spark.util.io.ChunkedByteBufferOutputStream.allocateNewChunkIfNeeded(ChunkedByteBufferOutputStream.scala:87)
  at org.apache.spark.util.io.ChunkedByteBufferOutputStream.write(ChunkedByteBufferOutputStream.scala:75)
  at java.base/java.io.OutputStream.write(OutputStream.java:127)
  at org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite.$anonfun$new$22(ChunkedByteBufferOutputStreamSuite.scala:127)
  at org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite$$Lambda$179/0x00000008011a75d8.apply(Unknown Source)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
{code}
Bug: Spark API to apply same function to multiple columns
Status: Open | Priority: Minor | Resolution: 
Created: 26/Sep/21 16:41
Affects Version/s: 2.4.8

Hi

My team and I have regularly need to apply the same function to multiple columns at once.

For example, we want to remove all non alphanumerical characters to each columns of our dataframes. 

When we hit this use case first, some people in my team were using this kind of code : 


{code:java}
val colListToClean = .... ## Generate some list, could be very long.
val dfToClean: DataFrame = ... ## This is the dataframe we want to clean
def cleanFunction(colName: String): Column = ... ## Write some function to manipulate column based on its name.
val dfCleaned = colListToClean.foldLeft(dfToClean)((df, colName) => df.withColumn(colName, cleanFunction(colName)){code}

This kind of code when applied on a large set of columns overloaded our driver (because a Dataframe is generated for each column to clean).

Based on this issue, we developed some code to add two functions : 


 * One to apply the same function to multiple columns
 * One to rename multiple columns based on a Map. 

 

I wonder if your ever ask your team to add such kind of API ? If you did, had you any kind of issue regarding the implementation ? If you didn't, is this any idea you could add to Spark ? 

Best regards, 

 

LvffY
Bug: Improve InSet NaN check generated code performance
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Sep/21 02:49
Affects Version/s: 3.3.0

Since Set can't check is NaN value is contained in current set.
With codegen, only when value set contains NaN then we have necessary to check if the value is NaN, or we just need t
o check is the Set contains the value.
Bug: Introduce a config variable for the incrementalCollects row batch size
Status: Open | Priority: Minor | Resolution: 
Created: 21/Sep/21 14:54
Affects Version/s: 3.1.2

After enabling *_spark.sql.thriftServer.incrementalCollects_* Thrift will execute queries in batches (as intended). Unfortunately the batch size cannot be configured as it seems to be hardcoded [here|https://github.com/apache/spark/blob/6699f76fe2afa7f154b4ba424f3fe048fcee46df/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java#L404]. It would be useful to configure that value to be able to adjust it to your environment.
Bug: Add an iterator method to Dataset
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 24/Sep/21 11:21
Affects Version/s: 3.3.0

The current org.apache.spark.sql.Dataset#toLocalIterator will submit multiple jobs for multiple partitions. 

In my case, I would like to collect all partition at once to save the job scheduling cost and also has an iterator to save the memory on deserialization (instead of deserialize all rows at once, I want only one row is deserialized during the iteration)

.
Bug: this is a spelling mistakes in running-on-yarn.md file where  JHS_POST should be JHS_HOST
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Sep/21 10:12
Affects Version/s: 3.2.0

{code:java}
NOTE: you need to replace <JHS_POST> and <JHS_PORT> with actual value

the JHS_POST should be JHS_HOST

{code}
Bug: Update user-facing catalog to adapt CatalogPlugin
Status: In Progress | Priority: Minor | Resolution: 
Created: 17/Sep/21 06:21
Affects Version/s: 3.3.0

At now the SparkSession.catalog always retuan a CatalogImpl with a SessionCatalog that is SparkSession.sessionState.catalog
{code:java}
@transient lazy val catalog: Catalog = new CatalogImpl(self)
{code}
{code:java}
private def sessionCatalog: SessionCatalog = sparkSession.sessionState.catalog
{code}
So we can do the action is just based the SessionCatalog, we could not do action based user-defined CatalogPlugin.
Bug: [K8S] Support write container stdout/stderr to file 
Status: In Progress | Priority: Minor | Resolution: 
Created: 17/Sep/21 17:56
Affects Version/s: 3.1.2

Currently, executor and driver pod only redirect stdout/stderr. If users want to sidecar logging agent to send stdout/stderr to external log storage,  only way is to change entrypoint.sh, which might break compatibility with community version.

We should support this feature, and this feature could be enabled by spark config. Related spark configs are:
|Key|Default|Desc|
|Spark.kubernetes.logToFile.enabled|false|Whether to write executor/driver stdout/stderr as log file|
|Spark.kubernetes.logToFile.path|/var/log/spark|The path to write executor/driver stdout/stderr as log file|
Bug: AQE dynamic join selection should not apply to non-equi join
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Sep/21 14:36
Affects Version/s: 3.3.0

Currently `DynamicJoinSelection` has two features: 1.demote broadcast hash join, and 2.promote shuffled hash join. Both are achieved by adding join hint in query plan, and only works for equi join. However the rule is matching with `Join` operator now - [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/DynamicJoinSelection.scala#L71,] so it would add hint for non-equi join by mistake.
Bug: Wrong order of variable initialization may lead to incorrect behavior
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Sep/21 12:05
Affects Version/s: 3.1.2

Incorrect order of variable initialization may lead to incorrect behavior, Related code: [TorrentBroadcast.scala|https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala#L94] , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.

Supplement:

Snippet 1:
{code:java}
class Broadcast {
  def setConf(): Unit = {
    checksumEnabled = true
  }
  setConf()
  var checksumEnabled = false
}

println(new Broadcast().checksumEnabled){code}
output:
{code:java}
false{code}
Snippet 2:
{code:java}
class Broadcast {
  var checksumEnabled = false
  def setConf(): Unit = {
    checksumEnabled = true
  }
  setConf()
}

println(new Broadcast().checksumEnabled){code}
output: 
{code:java}
true{code}
Bug: Drop type ignores from numpy imports
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Sep/21 21:09
Affects Version/s: 3.3.0

Currently we use {{type: ingore[import]}} on all numpy imports ‒ this was necessary because numpy didn't provide annotations at the time when we added stubs to PySpark.

Since numpy 1.20 (https://github.com/numpy/numpy/releases/tag/v1.20.0) numpy is PEP 561 compatible and these ignores are no longer necessary (current numpy version is 1.21).
Bug: RDD.toDF should be synchronized with dispatched variants of SparkSession.createDataFrame
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Sep/21 22:21
Affects Version/s: 3.3.0

There are some variants that are supported:
 * Providing a schema as a {{str}} object for {{RDD[RowLike]}} objects
 * Providing a schema as a {{Tuple[str, ...]}} names
 * Calling {{toDF}} on {{RDD}} of atomic values, when schema of {{str}} or {{AtomicType}} is provided.
Bug: Ignore duplicated join keys when building relation for SEMI/ANTI shuffle hash join
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Sep/21 18:59
Affects Version/s: 3.3.0

For LEFT SEMI and LEFT ANTI hash equi-join without extra join condition, we only need to keep one row per unique join key(s) inside hash table (`HashedRelation`) when building the hash table. This can help reduce the size of hash table of join.
Bug: Using the verbose parameter in yarn mode would cause application submission failure
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Sep/21 04:12
Affects Version/s: 3.1.2

If we submit the spark application with the --verbose parameter in yarn mode, we will get the following exception:
{code:java}
Exception in thread "main" java.lang.IllegalArgumentException: Unknown/unsupported param List(--verbose)Exception in thread "main" 
{code}
I check the code, found SparkSubmit invoke YarnClusterApplication with --verbose arguments, however ClientArguments used by YarnClusterApplication don't support that argument by now, as a result, IllegalArgumentException is thrown in ClientArguments. I think we can support --verbose in YARN mode to keep consistency with other module
Bug: When SparkContext is stopped, metrics system should be flushed after listeners have finished processing
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Sep/21 08:39
Affects Version/s: 2.3.2

In current implementation, when {{SparkContext.stop()}} is called, {{metricsSystem.report()}} is called before {{listenerBus.stop()}}. In this case, if some listener is producing some metrics, they would never reach sink.

Background:
We have some ingestion jobs in Spark Structured Streaming. To monitor them, collect some metrics like number of input rows, trigger time etc. from {{QueryProgressEvent}} received via {{StreamingQueryListener}}. These metrics are then pushed to DB by custom sinks registered in {{MetricsSystem}}. We noticed that these metrics are lost occasionally for last batch.
Bug: Building by "./build/mvn" may be stuck on MacOS
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Sep/21 09:19
Affects Version/s: 3.2.0

Command "./build/mvn" will be stuck on my MacOS 11.4. Because it is using error java home. On my mac, "/usr/bin/java" is a real file instead of a symbolic link, so the java home is set to path "/usr", and lead the launched maven process stuck with this error java home.
Bug: guava version mismatch with hadoop-aws
Status: In Progress | Priority: Minor | Resolution: 
Created: 27/Sep/21 17:33
Affects Version/s: 3.1.2

When use hadoop-aws 3.2 with spark 3.0, got below error. This is caused by guava version mismatch as hadoop used guava 27.0-jre while spark used 14.0.1.

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
 at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:742)
 at org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys(S3AUtils.java:712)
 at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:559)
 at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:264)
 at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
 at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
 at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
 at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1853)
 at org.apache.spark.deploy.history.EventLogFileWriter.<init>(EventLogFileWriters.scala:60)
 at org.apache.spark.deploy.history.SingleEventLogFileWriter.<init>(EventLogFileWriters.scala:213)
 at org.apache.spark.deploy.history.EventLogFileWriter$.apply(EventLogFileWriters.scala:181)
 at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:66)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:584)
 at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2588)
 at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:937)
 at scala.Option.getOrElse(Option.scala:189)
 at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:931)
 at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)
 at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
 at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:944)
 at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
 at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
 at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
 at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1023)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1032)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Bug: Unable to build Spark on Azure DevOps with ubuntu-latest
Status: Reopened | Priority: Minor | Resolution: 
Created: 08/Sep/21 11:38
Affects Version/s: 2.4.8

Hello

With my team we're currently triying to set up some test environment by using Spark on kubernetes.

For this purpose, we're following your (great) documentation to [build spark|https://spark.apache.org/docs/2.4.8/building-spark.html] and [build spark docker images|https://spark.apache.org/docs/latest/running-on-kubernetes.html#docker-images]. 

To make our build, we're using Azure DevOps.

I don't know if it's a known bug or requirements I didn't see but I found that I couldn't build spark on the Azure agent *ubuntu-latest* which I trust to be *ubuntu-20.04*. The exact same build works on *ubuntu-18.04*. 

My maven build always failed on building *spark-unsafe_2.11* with the following error :


{code:java}
[error] warning: [options] bootstrap class path not set in conjunction with -source 8 
[error] /home/vsts/work/1/s/spark/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java:25: error: cannot find symbol [error] import sun.misc.Cleaner; 
[error] ^ 
[error] symbol: class Cleaner 
[error] location: package sun.misc 
[error] /home/vsts/work/1/s/spark/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java:172: error: cannot find symbol 
[error] Cleaner cleaner = Cleaner.create(buffer, () -> freeMemory(memory)); 
[error] ^ 
[error] symbol: class Cleaner [error] location: class Platform 
[error] /home/vsts/work/1/s/spark/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java:172: error: cannot find symbol 
[error] Cleaner cleaner = Cleaner.create(buffer, () -> freeMemory(memory)); 
[error] ^ 
[error] symbol: variable Cleaner 
[error] location: class Platform 
[error] 3 errors 
[error] 1 warning 
[error] Compile failed at Sep 8, 2021 10:37:02 AM [1.126s]{code}
 

Please tell me if I miss anything, 

Best regards
Bug: DPP: Don't insert redundant filters in case static partition pruning can be done
Status: In Progress | Priority: Minor | Resolution: 
Created: 21/Sep/21 20:02
Affects Version/s: 3.1.2

Don't insert dynamic partition pruning filters in case the filters already referred statically. In case the filtering predicate on dimension table is in joinKey, no need to insert DPP filter in that case.

DPP is not required in this Sample query:
{code:java}
SELECT f.date_id, f.pid, f.sid FROM
 (select date_id, product_id as pid, store_id as sid from fact_stats) as f
 JOIN dim_stats s
 ON f.sid = s.store_id WHERE s.store_id = 3{code}