Bug: java.lang.IllegalArgumentException Related to Prometheus
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/21 21:19
Affects Version/s: 3.1.1

This issue is similar to https://issues.apache.org/jira/browse/SPARK-35237?focusedCommentId=17340723&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17340723. We receive the Following warning continuously:

 

21:00:26.277 [rpc-server-4-2] WARN  o.a.s.n.s.TransportChannelHandler - Exception in connection from /10.198.3.179:51184java.lang.IllegalArgumentException: Too large frame: 5135603447297303916 at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148) at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:98) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source)

 

Below are other details related to prometheus and my findings. Please SCROLL DOWN to see the details:

 
{noformat}
Prometheus Scrape Configuration
===============================
- job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2

tcptrack command output in spark3 pod
======================================
10.198.22.240:51258  10.198.40.143:7079  CLOSED 10s 0 B/s
10.198.22.240:51258  10.198.40.143:7079  CLOSED 10s 0 B/s
10.198.22.240:50354  10.198.40.143:7079  CLOSED 40s 0 B/s
10.198.22.240:33152  10.198.40.143:4040  ESTABLISHED 2s 0 B/s
10.198.22.240:47726  10.198.40.143:8090  ESTABLISHED 9s 0 B/s

10.198.22.240 = prometheus pod 

ip10.198.40.143 = testpod ip 

Issue
======
Though the scrape config is expected to scrape on port 8090. I see prometheus tries to initiate scrape on ports like 7079, 7078, 4040, etc on
the spark3 pod and hence the exception in spark3 pod. But is this really a prometheus issue or something at spark side? We don't see any such exception in any of the other pods. All our pods including spark3 are annotated with:

annotations:
   prometheus.io/port: "8090"
   prometheus.io/scrape: "true"

We get the metrics and everything fine just extra warning for this exception.{noformat}