ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
Image-Text-to-Text • 27B • Updated • 403k • 480
NH
Nhebo
·
AI & ML interests
None yet
Recent Activity
liked a model 2 days ago
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF liked a model 5 days ago
peculiar-ragdoll/Qwen-Sharp-Chat-Templates liked a model 14 days ago
orcarouter/Qwen3.8-27B-Uncensored-GGUFOrganizations
None yet
liked a
model 2 days ago
liked a
model 5 days ago
liked a
model 14 days ago
liked a
model 20 days ago
liked a
model 21 days ago
liked a
model 22 days ago
liked a
model 24 days ago
liked a
model about 2 years ago
liked 2
models over 2 years ago
reacted to santiviquez's post with 👍 over 2 years ago
Post
Where I work, we are obsessed with what happens to a model's performance after it has been deployed. We call this post-deployment data science.
Let me tell you about a post-deployment data science algorithm that we recently developed to measure the impact of Concept Drift on a model's performance.
How can we detect Concept Drift? 🤔
All ML models are designed to do one thing: learning a probability distribution in the form of P(y|X). In other words, they try to learn how to model an outcome 'y' given the input variables 'X'. 🧠
This probability distribution, P(y|X), is also called Concept. Therefore, if the Concept changes, the model may become invalid.
❓But how do we know if there is a new Concept in our data?
❓Or, more important, how do we measure if the new Concept is affecting the model's performance?
💡 We came up with a clever solution where the main ingredients are a reference dataset, one where the model's performance is known, and a dataset with the latest data we would like to monitor.
👣 Step-by-Step solution:
1️⃣ We start by training an internal model on a chunk of the latest data. ➡️ This allows us to learn the new possible Concept presented in the data.
2️⃣ Next, we use the internal model to make predictions on the reference dataset.
3️⃣ We then estimate the model's performance on the reference dataset, assuming the model's predictions on the monitoring data as ground truth.
4️⃣ If the estimated performance of the internal model and the actual monitored model are very different, we then say that there has been a Concept Drift.
To quantify how this Concept impacts performance, we subtract the actual model's performance on reference from the estimated performance and report a delta of the performance metric. ➡️ This is what the plot below shows. The change of the F1-score due to Concept drift! 🚨
This process is repeated for every new chunk of data that we get. 🔁
Let me tell you about a post-deployment data science algorithm that we recently developed to measure the impact of Concept Drift on a model's performance.
How can we detect Concept Drift? 🤔
All ML models are designed to do one thing: learning a probability distribution in the form of P(y|X). In other words, they try to learn how to model an outcome 'y' given the input variables 'X'. 🧠
This probability distribution, P(y|X), is also called Concept. Therefore, if the Concept changes, the model may become invalid.
❓But how do we know if there is a new Concept in our data?
❓Or, more important, how do we measure if the new Concept is affecting the model's performance?
💡 We came up with a clever solution where the main ingredients are a reference dataset, one where the model's performance is known, and a dataset with the latest data we would like to monitor.
👣 Step-by-Step solution:
1️⃣ We start by training an internal model on a chunk of the latest data. ➡️ This allows us to learn the new possible Concept presented in the data.
2️⃣ Next, we use the internal model to make predictions on the reference dataset.
3️⃣ We then estimate the model's performance on the reference dataset, assuming the model's predictions on the monitoring data as ground truth.
4️⃣ If the estimated performance of the internal model and the actual monitored model are very different, we then say that there has been a Concept Drift.
To quantify how this Concept impacts performance, we subtract the actual model's performance on reference from the estimated performance and report a delta of the performance metric. ➡️ This is what the plot below shows. The change of the F1-score due to Concept drift! 🚨
This process is repeated for every new chunk of data that we get. 🔁
liked a
dataset about 3 years ago
liked 2
models over 3 years ago
liked 3
Spaces over 3 years ago
liked 2
models over 3 years ago
