Repository: apache/skywalking (language: python)

Issue:
[Feature] [OAP] Consider removing metrics reload in persistent
### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.


### Description

In 9.3.0, we enhanced the cache mechanism in #10021 and verified through #10025. The `ID` read by the persistent worker at the minute dimensionality now has 60% cached. Only the first bulk of every minute would require `ID` read.

When we look deeper into the logic, we would find out actually that, **from the minute dimensionality, this `ID` read is not required**.
We are concerned that no metric in the cache doesn't mean there is no metric in the database. Especially when 
1. The timestamp is not synced in the cluster, so, timestamps of telemetry are not ordered by the time series.
2. OAP is booting/rebooting, and the cache is cold.

About <1>, we don't expect this anymore. Our TTL and metric/topo analysis are all relying on timestamps generally synced. It doesn't have to be synced in the `ms` level, but at least with only a 3-5s gap.

About <2>, we only should try to load metrics from DB in the 1 minute after rebooting, considering the assumption about time synced in the <1>. So, the metrics would overlap existing metrics generated in one booting period. 
There is little chance we faced data conflicts, even if we faced them, we just generate metrics at the booting minute inaccurate. In the best practice, we could keep loading metrics from the database when metrics timestamps are before the **OAP started timestamp** as a fail-safe.

Regarding hour and day dimensionalities, there is nothing different. We just should keep `loading metrics from database` when the hour/day time bucket before **OAP started hour / day**.

@hanahmily @kezhenxu94 @wankai123 PTAL. Considering 9.3.0 releasing soon, I don't want to take a risk to change for now.
But the theory should be correct, please help on rechecking.

### Use case

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)



Discussion / hints:
FYI @apache/skywalking-committers if you have interests in SkyWalking OAP kernel logic.
Ideally, due to this enhancement, the cached metrics should go closing to 95%+, and highly reduce the IOPS of the database(read part). 
The OAP would only read from database when CLI/UI applies for query metrics/traces/logs.
I'm delighted to see this will happen. BanyanDB is based on the idea that we read less than write. Reducing reading operations will improve interactive performance technically. 

The timeline is also suitable for me. The BanyanDB storage plugin of OAP needs more enhancements from my recent works on the showcase env. We could make it happen in the next iteration. 
This is a great idea to reduce the load on the storage side! I think we can start with the minute dimensionality, it's the most frequent. And for the hour/day dimensionality it's difficult to estimate the expiration time of the cache.

Fix patch:
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 4e7bd5c6b4f3..d1a9e01160cb 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -2,14 +2,12 @@
 
 #### Project
 
-
 #### OAP Server
 
+* Add `ServerStatusService` in the core module to provide a new way to expose booting status to other modules.
 
 #### UI
 
-
 #### Documentation
 
-
 All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/160?closed=1)
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModule.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModule.java
index 5c67c10a5344..66d9934e97da 100755
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModule.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModule.java
@@ -33,6 +33,7 @@
 import org.apache.skywalking.oap.server.core.profiling.ebpf.EBPFProfilingMutationService;
 import org.apache.skywalking.oap.server.core.profiling.ebpf.EBPFProfilingQueryService;
 import org.apache.skywalking.oap.server.core.profiling.trace.ProfileTaskMutationService;
+import org.apache.skywalking.oap.server.core.profiling.trace.ProfileTaskQueryService;
 import org.apache.skywalking.oap.server.core.query.AggregationQueryService;
 import org.apache.skywalking.oap.server.core.query.AlarmQueryService;
 import org.apache.skywalking.oap.server.core.query.BrowserLogQueryService;
@@ -41,7 +42,6 @@
 import org.apache.skywalking.oap.server.core.query.MetadataQueryService;
 import org.apache.skywalking.oap.server.core.query.MetricsMetadataQueryService;
 import org.apache.skywalking.oap.server.core.query.MetricsQueryService;
-import org.apache.skywalking.oap.server.core.profiling.trace.ProfileTaskQueryService;
 import org.apache.skywalking.oap.server.core.query.RecordQueryService;
 import org.apache.skywalking.oap.server.core.query.TagAutoCompleteQueryService;
 import org.apache.skywalking.oap.server.core.query.TopNRecordsQueryService;
@@ -52,6 +52,7 @@
 import org.apache.skywalking.oap.server.core.server.GRPCHandlerRegister;
 import org.apache.skywalking.oap.server.core.server.HTTPHandlerRegister;
 import org.apache.skywalking.oap.server.core.source.SourceReceiver;
+import org.apache.skywalking.oap.server.core.status.ServerStatusService;
 import org.apache.skywalking.oap.server.core.storage.model.IModelManager;
 import org.apache.skywalking.oap.server.core.storage.model.ModelCreator;
 import org.apache.skywalking.oap.server.core.storage.model.ModelManipulator;
@@ -73,6 +74,7 @@ public CoreModule() {
     public Class[] services() {
         List<Class> classes = new ArrayList<>();
         classes.add(ConfigService.class);
+        classes.add(ServerStatusService.class);
         classes.add(DownSamplingConfigService.class);
         classes.add(NamingControl.class);
         classes.add(IComponentLibraryCatalogService.class);
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java
index 2f96358ae61d..429830cfab75 100755
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java
@@ -81,6 +81,7 @@
 import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
 import org.apache.skywalking.oap.server.core.source.SourceReceiver;
 import org.apache.skywalking.oap.server.core.source.SourceReceiverImpl;
+import org.apache.skywalking.oap.server.core.status.ServerStatusService;
 import org.apache.skywalking.oap.server.core.storage.PersistenceTimer;
 import org.apache.skywalking.oap.server.core.storage.StorageException;
 import org.apache.skywalking.oap.server.core.storage.model.IModelManager;
@@ -240,6 +241,7 @@ public void prepare() throws ServiceNotProvidedException, ModuleStartException {
         httpServer.initialize();
 
         this.registerServiceImplementation(ConfigService.class, new ConfigService(moduleConfig));
+        this.registerServiceImplementation(ServerStatusService.class, new ServerStatusService(getManager()));
         this.registerServiceImplementation(
             DownSamplingConfigService.class, new DownSamplingConfigService(moduleConfig.getDownsampling()));
 
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/BootingStatus.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/BootingStatus.java
new file mode 100644
index 000000000000..4de30fcff836
--- /dev/null
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/BootingStatus.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.status;
+
+import lombok.AccessLevel;
+import lombok.Getter;
+import lombok.Setter;
+
+/**
+ * Booting status indicate whether the current server starts successfully.
+ */
+@Getter
+@Setter(AccessLevel.PACKAGE)
+public class BootingStatus {
+    /**
+     * The status of OAP is fully booted successfully.
+     */
+    private boolean isBooted = false;
+    /**
+     * The uptime in milliseconds if {@link #isBooted} is true;
+     */
+    private long uptime = 0;
+}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java
new file mode 100644
index 000000000000..27a3b328dfa5
--- /dev/null
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.status;
+
+import lombok.RequiredArgsConstructor;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.library.module.Service;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;
+
+/**
+ * The server status service provides the indicators for the current server status.
+ * Notice, this should not be treated as a kind of health checker or self telemetry.
+ * For more, this helps modules to be aware of current OAP server status.
+ *
+ * @since 9.4.0
+ */
+@RequiredArgsConstructor
+public class ServerStatusService implements Service {
+    private final ModuleManager manager;
+    private BootingStatus bootingStatus = new BootingStatus();
+
+    public void bootedNow(long uptime) {
+        bootingStatus.setBooted(true);
+        bootingStatus.setUptime(uptime);
+        manager.find(TelemetryModule.NAME)
+               .provider()
+               .getService(MetricsCreator.class)
+               .createGauge("uptime", "oap server start up time", MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE)
+               // Set uptime to second
+               .setValue(uptime / 1000d);
+    }
+
+}
diff --git a/oap-server/server-starter/src/main/java/org/apache/skywalking/oap/server/starter/OAPServerBootstrap.java b/oap-server/server-starter/src/main/java/org/apache/skywalking/oap/server/starter/OAPServerBootstrap.java
index 2b8f4b8e9990..f1f2a2d37dd4 100644
--- a/oap-server/server-starter/src/main/java/org/apache/skywalking/oap/server/starter/OAPServerBootstrap.java
+++ b/oap-server/server-starter/src/main/java/org/apache/skywalking/oap/server/starter/OAPServerBootstrap.java
@@ -19,14 +19,13 @@
 package org.apache.skywalking.oap.server.starter;
 
 import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.CoreModule;
 import org.apache.skywalking.oap.server.core.RunningMode;
+import org.apache.skywalking.oap.server.core.status.ServerStatusService;
 import org.apache.skywalking.oap.server.core.version.Version;
 import org.apache.skywalking.oap.server.library.module.ApplicationConfiguration;
 import org.apache.skywalking.oap.server.library.module.ModuleManager;
 import org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader;
-import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
-import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
-import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;
 
 /**
  * Starter core. Load the core configuration file, and initialize the startup sequence through {@link ModuleManager}.
@@ -43,12 +42,10 @@ public static void start() {
             ApplicationConfiguration applicationConfiguration = configLoader.load();
             manager.init(applicationConfiguration);
 
-            manager.find(TelemetryModule.NAME)
+            manager.find(CoreModule.NAME)
                    .provider()
-                   .getService(MetricsCreator.class)
-                   .createGauge("uptime", "oap server start up time", MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE)
-                   // Set uptime to second
-                   .setValue(System.currentTimeMillis() / 1000d);
+                   .getService(ServerStatusService.class)
+                   .bootedNow(System.currentTimeMillis());
 
             log.info("Version of OAP: {}", Version.CURRENT);
 


Test patch:
diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/CoreModuleTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/CoreModuleTest.java
index 4d69c6996616..e62c4ad3d67b 100644
--- a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/CoreModuleTest.java
+++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/CoreModuleTest.java
@@ -26,6 +26,6 @@ public class CoreModuleTest {
     public void testOpenServiceList() {
         CoreModule coreModule = new CoreModule();
 
-        Assert.assertEquals(37, coreModule.services().length);
+        Assert.assertEquals(38, coreModule.services().length);
     }
 }